🔤Tokenization & Vocabulary Design

Tokenization is the discrete interface between raw text/bytes and a transformer. The model never sees “characters” or “words” — it sees token IDs from a vocabulary $\mathcal V$.

---

## 1) Tokenization is a segmentation into vocabulary items

A tokenizer maps a string/byte sequence $x$ into tokens $t_1,\dots,t_n$:

Then each token becomes an integer ID $\mathrm{id}(t_i) \in \{0,\dots,|\mathcal V|-1\}$ that indexes the embedding table.

---

## 2) Unigram tokenization as maximum-likelihood segmentation (SentencePiece-style)

Unigram models treat tokenization itself as probabilistic inference: choose the segmentation that maximizes token prior likelihood:

This is a Viterbi / dynamic programming problem in practice: you search over segmentations and pick the highest-probability path.

---

## 3) BPE (byte-pair encoding) builds the vocabulary by merges

BPE starts from base symbols (characters or bytes), then repeatedly merges the most frequent adjacent pair:

Each merge increases vocabulary size (one new token) and usually decreases sequence length on texts where that pair is common.

---

## 4) Vocabulary size is a real model parameter (embedding + softmax head)

Token IDs index the embedding matrix, and logits are produced over the same vocabulary:

So parameters scale roughly like:

Tradeoff: larger vocab can mean fewer tokens (cheaper context / KV cache), but larger embedding + output layers (more parameters / memory).

---

### Why this is a “foundation” Tokenization silently shapes: - what patterns become “single atoms” (code, whitespace, common substrings), - how expensive prompts are (tokens per character), - multilingual/Unicode behavior (bytes vs scripts vs normalization), - and even security surfaces (invisible characters, homoglyphs, normalization).