Skip to content
AI LAB
01
Tokens
02
Prediction & Sampling
03
Embeddings
04
RAG
05
Evals
1/5
explorer
engineer
LAB 01 / TOKENISATION
Models don't read words.
Type your own sentence below, or pick an example, then Tokenize.
The model sees your sentence differently.
Tokenize
41/400
Normal English
Long / unusual word
Code
Numbers
Emoji
Punctuation
Next
02 Prediction & Sampling
How does the model choose the next token?