Tokenization splits text into tokens, pieces drawn from a fixed vocabulary, and maps each token to an integer ID. Prompt prices and context limits are counted in tokens, not characters, and the computation and latency of generation grow with the number of tokens.
A newer notebook covers this concept. This older page keeps its summary, math and sources.
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you move a control in the demo, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the demo shows with the key equation and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueContinue in the newer notebook.
The newer notebook covers the same concept and has an interactive demo.
Why It Matters for Modern Models
Tokenization splits text into tokens, pieces drawn from a fixed vocabulary, and maps each token to an integer ID. Prompt prices and context limits are counted in tokens, not characters, and the computation and latency of generation grow with the number of tokens.
Tokenizer design changes what a model finds easy: code, numbers and non-English text can be compact or split into many pieces depending on where subword boundaries fall.
Long-context engineering depends on token counts: a "128k context" holds 128k tokens, and the tokenizer decides how much text that is.
Vocabulary size is an architectural choice: the embedding and output layers grow with |V|, trading parameters for shorter sequences.
Unicode edge cases (normalization, invisible characters) affect reliability and safety: two strings that look identical can tokenize differently.
Questions to Think About
BPE merges whatever is frequent in its training text; why do rare strings such as identifiers and long numbers break into many tokens?
Why does unigram tokenization choose the most probable segmentation rather than the one with the longest tokens?
Byte-level tokenization never fails on unseen characters, so why does it make non-Latin scripts and emoji more expensive?
How can Unicode normalization (NFC versus NFKC, zero-width characters, non-breaking spaces) change token boundaries and cost without changing what the text looks like?
Interactive Demo
Core Math
If you prefer intuition first, try the demo above, then come back for the math.
Key Equation
t^1:n=argt:concat(t)=xmaxi=1∑nlogp(ti)
Tokenization is the discrete interface between raw text (or bytes) and a transformer. The model never sees characters or words directly; it sees token IDs from a vocabulary V.
1) Tokenization is a segmentation into vocabulary items
A tokenizer maps a string or byte sequence x into tokens t1,…,tn:
x=concat(t1,…,tn),ti∈V
Each token then becomes an integer ID id(ti)∈{0,…,∣V∣−1} that indexes the embedding table.
2) Unigram tokenization as maximum-likelihood segmentation
The unigram language model used in SentencePiece treats segmentation as probabilistic inference: it chooses the segmentation with the highest total log-probability under per-token probabilities p(t) (Kudo, 2018):
t^1:n=argt:concat(t)=xmaxi=1∑nlogp(ti)
In practice this is solved with the Viterbi algorithm, a dynamic program over all possible segmentations.
3) BPE (byte-pair encoding) builds the vocabulary by merges
BPE starts from base symbols (characters or bytes), then repeatedly merges the most frequent adjacent pair (Sennrich et al., 2016):
(a,b)∗=arg(a,b)maxcount(ab)⇒ab→ab
Each merge adds one token to the vocabulary and usually shortens texts in which that pair is common.
4) Vocabulary size is a real model parameter (embedding and output layer)
Token IDs index the embedding matrix, and the output layer produces logits over the same vocabulary:
Wembed∈R∣V∣×d,Wout∈R∣V∣×d
So the token-dependent parameters number about
paramstoken≈2∣V∣d
or ∣V∣d when the input and output matrices share their weights.
Trade-off: a larger vocabulary usually means fewer tokens per text (cheaper context and a smaller KV cache) but larger embedding and output layers.
What tokenization affects
Tokenization quietly shapes:
which patterns become single units (code keywords, whitespace runs, common substrings),
how expensive a prompt is (tokens per character),
how multilingual and Unicode text is handled (bytes versus scripts versus normalization),
and some security issues (invisible characters, look-alike characters, normalization).
Key Papers
Neural Machine Translation of Rare Words with Subword Units
Token count grows: where Tokenization breaks down, taken up in Long Context. Open Long Context
Context length is measured in tokens, not characters. Text that splits into many small pieces (rare Unicode, code identifiers, byte-level fallbacks) inflates the token count and uses up the context window and the key-value cache faster than expected.
Unseen words: where Tokenization breaks down, taken up in Maximum Likelihood. Open Maximum Likelihood
A word-level vocabulary has no token for an unseen word, so the model cannot assign it any probability. Subword methods such as byte-pair encoding (Sennrich et al., 2016), especially with a fallback to bytes, remove that problem, but rare strings are split into many pieces whose probabilities are estimated from little data.
Token units shift: where Scaling Laws breaks down, taken up on this page. Open Scaling Laws
Scaling laws report cross-entropy per token, but tokenizer changes shift what "a token" means. Curves can look better/worse unless normalized (bits-per-byte).
Where else does the same idea appear?
Codebook sparsity: an analogy between Tokenization and Sparse Autoencoders. Open Sparse Autoencoders
Tokenization represents text by selecting items from a learned codebook (vocab). SAEs represent activations by selecting items from a learned codebook (features)—both are "dictionary + sparse code" stories.
Compression tradeoff: an analogy between Tokenization and Superposition. Open Superposition
Tokenizers compress frequent patterns into single tokens but fragment rare strings into many pieces. Superposition compresses many features into limited dimensions but creates interference—same capacity vs ambiguity tradeoff.
Vision transformers cut an image into patches and treat each patch embedding like a token embedding, so the same transformer can process images and text (Dosovitskiy et al., 2021). Unlike text tokens, patches are not drawn from a fixed vocabulary.
Compression efficiency: an analogy between Tokenization and Scaling Laws. Open Scaling Laws
Vocabulary size changes how many tokens a text takes and how many embedding parameters a model has. Tao et al. (2024) find that the compute-optimal vocabulary grows with model size; they estimate that Llama 2 70B would have been better served by a vocabulary of at least 216,000 tokens than by its 32,000.
Viterbi segmentation: the same technique appears in Maximum Likelihood. Open Maximum Likelihood
Unigram tokenization chooses the max-prob segmentation by maximizing ∑ log p(tᵢ) under concat constraints. That is MAP/MLE-style inference solved by dynamic programming (a Viterbi-like trick).
Everything as a sequence: an analogy between Tokenization and Multimodal Pretraining. Open Multimodal Pretraining
Many multimodal models turn every input into a sequence of vectors that a transformer can process: image patches, pieces of audio, and text tokens. The sequence model is shared; only the way each input is cut into pieces differs.
Which ideas are its mathematical duals?
Tokens ↔ Codelength: Learning Theory is its mathematical dual. Open Learning Theory
A tokenizer is a codebook: vocab size and token counts jointly define a compression scheme. MDL provides the "dual" way to reason about why changing vocab changes sequence length, capacity, and generalization behavior.