Attention & Transformers

Tokenizer Training from Raw Text

How raw text becomes a learned subword vocabulary: weighted pre-tokens, pair counts, greedy BPE merges, and the merge-order consequences learners can inspect.

status: reviewimportance: criticaldifficulty 3/5math: undergraduateread: 15mlive demo

Back to Attention & Transformers Next: Raw Text to Packed Causal-LM Examples (review)

Concept Structure

Tokenizer Training from Raw Text

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

2prerequisites

3next concepts

3related links

Learner Contract

What this page should let you do.

You are here becauseHow raw text becomes a learned subword vocabulary: weighted pre-tokens, pair counts, greedy BPE merges, and the merge-order consequences learners can inspect.

This Attention & Transformers concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

Before thisTokenization & Vocabulary Design

2 prerequisites listed; refresh them before leaning on the math or code.

Tokenization & Vocabulary Design N-gram Language Models (review)

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextRaw Text to Packed Causal-LM Examples (review)

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to Raw Text to Packed Causal-LM Examples (review)

Claim/source review status

Substantive review recorded

2/2 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.

Claims2/2 reviewed

Sources5 cited

Codeattached

Demolive

Reviewed2026-07-03

Updatedpage 2026-07-03

Learning item flow

4/4 sections readyAsk about this Research room

ConceptTokenizer Training from Raw TextAttention & Transformers EquationTokenizer Training from Raw Text equation 1Exact equation CodeTokenizer Training from Raw Text runnable code 1Exact runnable code DemoTokenizer Training from Raw Text interactive demoInteractive demo ClaimA BPE tokenizer can be trained by converting raw text to weighted pre...Exact claim check SourceNeural Machine Translation of Rare Words with Subword UnitsExact source

ConceptTokenizer Training from Raw TextAttention & Transformers

5 sources attachedLocal snapshot ready

concept:attention-transformers/tokenizer-training-raw-text

Codenearby Predictbefore reveal Roomitem handoff

01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

A tokenizer is not just a lookup table someone typed by hand. A modern subword tokenizer is usually trained from text.

Start with raw documents. Normalize and pre-tokenize them into pieces such as words or byte sequences. Count how often those pieces appear. Then learn a vocabulary that compresses repeated local patterns without pretending every possible word deserves its own token.

BPE makes one beautifully blunt move:

Split each weighted pre-token into starting symbols.
Count adjacent symbol pairs across the weighted corpus.
Merge the most frequent pair.
Repeat until the vocabulary reaches the target size.

That means tokenizer training is a learned compression policy. The corpus decides which fragments become cheap to represent. If u g appears inside many high-frequency words, BPE may learn ug. If later h ug is frequent, it may learn hug. The merge list becomes an ordered program that later encodes new text.

This is the missing bridge between "tokenization exists" and "a causal LM trains on packed token IDs." Before the model sees a single batch, the tokenizer has already chosen the discrete alphabet for the learning problem.

Source spine: Sennrich, Haddow, and Birch, Neural Machine Translation of Rare Words with Subword Units; Kudo and Richardson, SentencePiece; Hugging Face, Byte-Pair Encoding tokenization and Tokenizers docs; Stanford CS336 Assignment 1: Basics.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt

Equation 1 $\mathcal C = \{(x_i, f_i)\}_{i=1}^m,$ Equation 2 $s_i^{(0)} = (c_1, c_2, \dots, c_{|x_i|}).$

Let a raw corpus be converted into weighted pre-tokens:

\mathcal C = \{(x_i, f_i)\}_{i=1}^m,

where $x_i$ is a pre-token such as hug and $f_i$ is its corpus frequency. In the toy lab, $x_i$ is split into characters:

s_i^{(0)} = (c_1, c_2, \dots, c_{|x_i|}).

At merge step $k$ , BPE counts every adjacent pair $(a,b)$ in the current symbolizations:

N_k(a,b) = \sum_{i=1}^m f_i \sum_{j=1}^{|s_i^{(k)}|-1} \mathbf 1[s_{i,j}^{(k)}=a \land s_{i,j+1}^{(k)}=b].

The next merge is the highest-count pair:

(a_k,b_k) = \arg\max_{(a,b)} N_k(a,b), \qquad a_k b_k \in \mathcal V_{k+1}.

Then every non-overlapping occurrence of $(a_k,b_k)$ is replaced by the new symbol $a_kb_k$ :

s_i^{(k+1)} = \operatorname{merge}(s_i^{(k)}, a_k, b_k).

After $K$ merges, the tokenizer stores the starting symbols and the ordered merge list:

\mathcal M = [(a_1,b_1), \dots, (a_K,b_K)].

Encoding a new pre-token applies those merges in order. The ordering matters: a later merge can only fire after earlier merges have created its left or right symbol. This is why pun may stay as p + un until the later rule p + un -> pun exists.

SentencePiece is an important counterpoint: it can train from raw sentences without assuming a language-specific word pre-tokenizer, and it supports both BPE and unigram language-model subword segmentation. This page stays with tiny BPE so the pair-count mechanics are visible.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt

Runnable code 1from collections import Counter corpus = {"hug": 10, "pug": 5, "pun": 12, "bun": 4, "hugs": 5...python

This small script mirrors the lab: count weighted adjacent pairs, merge the winner, and inspect the merge order. It is intentionally tiny rather than production-ready.

from collections import Counter

corpus = {"hug": 10, "pug": 5, "pun": 12, "bun": 4, "hugs": 5}

def pair_counts(vocab):
    counts = Counter()
    for symbols, freq in vocab.items():
        for pair in zip(symbols, symbols[1:]):
            counts[pair] += freq
    return counts

def apply_merge(symbols, pair):
    out, i = [], 0
    while i < len(symbols):
        if i + 1 < len(symbols) and (symbols[i], symbols[i + 1]) == pair:
            out.append(symbols[i] + symbols[i + 1])
            i += 2
        else:
            out.append(symbols[i])
            i += 1
    return tuple(out)

vocab = {tuple(word): freq for word, freq in corpus.items()}
merges = []

for _ in range(4):
    counts = pair_counts(vocab)
    best = max(counts, key=lambda pair: (counts[pair], pair))
    merges.append((best, counts[best]))
    vocab = {apply_merge(symbols, best): freq for symbols, freq in vocab.items()}

print("merges:", merges)
print("final vocab states:", vocab)

def encode(word, learned_merges):
    symbols = tuple(word)
    for pair, _ in learned_merges:
        symbols = apply_merge(symbols, pair)
    return symbols

print("pun before final:", encode("pun", merges[:3]))
print("pun after final:", encode("pun", merges))

Expected first merge: ("u", "g") -> "ug" with weighted count 20, because it appears inside hug, pug, and hugs.

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Prediction check: inspect the raw word frequencies, then predict which adjacent pair BPE should merge first and which probe word gets shorter only after the fourth merge.

The lab hides the pair-count table, merge timeline, vocabulary growth, and probe encodings until you commit. The goal is to feel tokenizer training as a sequence of auditable compression decisions, not a preprocessing footnote.

Live Concept Demo

Explore Tokenizer Training from Raw Text

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 3/5undergraduatecode-aligned

Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Tokenizer Training from Raw Text should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

How raw text becomes a learned subword vocabulary: weighted pre-tokens, pair counts, greedy BPE merges, and the merge-order consequences learners can inspect.

Prediction open01 / Intuition

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Tokenizer Training from Raw Text should make visible.

Visual Inquiry

Make the image answer a mathematical question

How raw text becomes a learned subword vocabulary: weighted pre-tokens, pair counts, greedy BPE merges, and the merge-order consequences learners can inspect.

4/4 stages readyLive demo connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Tokenizer Training from Raw Text easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

paper · 2015Neural Machine Translation of Rare Words with Subword UnitsSennrich, Haddow, and Birch

Primary BPE subword source. The paper motivates open-vocabulary subword units and describes BPE as repeatedly merging frequent symbol pairs.

Open source

paper · 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text ProcessingKudo and Richardson

Primary source for raw-sentence tokenizer training and for distinguishing BPE from unigram language-model subword segmentation.

Open source

course-notes · 2026Hugging Face LLM Course: Byte-Pair Encoding tokenizationHugging Face

Course source for a small BPE training walk-through over hug/pug/pun/bun/hugs word frequencies, pair counts, merge rules, and encoding.

Open source

documentation · 2026Hugging Face TokenizersHugging Face

Documentation-level source for production tokenizer components such as normalization, pre-tokenization, models, trainers, decoders, padding, and special-token processing.

Open source

course-notes · 2025Stanford CS336 Assignment 1: BasicsStanford CS336 course staff

Course-assignment source for positioning tokenizer implementation and raw text data as an early language-model-from-scratch object.

Open source

Claim Review

How raw text becomes a learned subword vocabulary: weighted pre-tokens, pair counts, greedy BPE merges, and the merge-order consequences learners can inspect.

Status2 substantive reviews recorded

Claims without a substantive review badge still need exact source-support review.

Sources5 references

sennrich-2015-bpe, kudo-2018-sentencepiece, hf-llm-course-bpe, hf-tokenizers-docs, cs336-assignment1-basics

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedA BPE tokenizer can be trained by converting raw text to weighted pre-tokens, initializing each pre-token as symbols, repeatedly counting adjacent symbol pairs, merging the most frequent pair, and reusing the learned merge order to encode new text.Claim metadata: source checked

Sennrich et al. support the BPE merge idea for open-vocabulary subword units; the Hugging Face course supports the tiny weighted-word pair-count example used in the demo; Hugging Face Tokenizers docs support the broader production pipeline terms; CS336 supports treating tokenizer implementation and raw datasets as part of an LLM-from-scratch route.

Sources: Neural Machine Translation of Rare Words with Subword Units, Hugging Face LLM Course: Byte-Pair Encoding tokenization, Hugging Face Tokenizers, Stanford CS336 Assignment 1: BasicsThe page teaches a tiny word-level, character-initialized BPE trainer. It does not implement byte-level GPT tokenization, Unicode normalization, regex pre-tokenization, special token policies, distributed training, or quality claims about a real production tokenizer.A bounded review summary is present; still check caveats and exact reference scope.

Checked Sennrich et al., Kudo/Richardson, Hugging Face BPE course/docs, and Stanford CS336 assignment material. The learner-facing page remains review-status pending GPT Pro publication critique.

Reviewer: codex-local-primary-source-audit; reviewed 2026-07-03

source-span-sennrich-2015-bpe source-span-kudo-2018-sentencepiece source-span-hf-llm-course-bpe source-span-hf-tokenizers-docs source-span-cs336-assignment1-basics Equation 1 Runnable code 1 Interactive demo

Substantively reviewedSentencePiece is a useful counterpoint because it can train subword models directly from raw sentences and supports both BPE and unigram language-model tokenization.Claim metadata: source checked

Kudo and Richardson describe SentencePiece as language independent, raw-sentence trainable, and supporting BPE plus unigram language-model subword segmentation.

Sources: SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text ProcessingThe interactive demo is BPE-only; unigram segmentation is queued as a downstream comparison concept.A bounded review summary is present; still check caveats and exact reference scope.

Checked Kudo and Richardson's SentencePiece abstract/source record for language-independent tokenization, direct raw-sentence subword model training, and BPE/unigram support. The page uses this only as a scoped counterpoint and does not implement unigram segmentation.

Reviewer: codex-local-primary-source-audit; reviewed 2026-07-03

source-span-kudo-2018-sentencepiece

Source support candidates

paper 2015Neural Machine Translation of Rare Words with Subword Units

Primary BPE subword source. The paper motivates open-vocabulary subword units and describes BPE as repeatedly merging frequent symbol pairs.

paper 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Primary source for raw-sentence tokenizer training and for distinguishing BPE from unigram language-model subword segmentation.

course-notes 2026Hugging Face LLM Course: Byte-Pair Encoding tokenization

Course source for a small BPE training walk-through over hug/pug/pun/bun/hugs word frequencies, pair counts, merge rules, and encoding.

documentation 2026Hugging Face Tokenizers

Documentation-level source for production tokenizer components such as normalization, pre-tokenization, models, trainers, decoders, padding, and special-token processing.

Mechanism checks

Equation 1

\mathcal C = \{(x_i, f_i)\}_{i=1}^m,

Equation 2

s_i^{(0)} = (c_1, c_2, \dots, c_{|x_i|}).

Runnable code 1from collections import Counter corpus = {"hug": 10, "pug": 5, "pun": 12, "bun": 4, "hugs": 5...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

How raw text becomes a learned subword vocabulary: weighted pre-tokens, pair counts, greedy BPE merges, and the merge-order consequences learners can inspect.

Readiness0/3 checks ready

Predict

Before touching the demo, predict one visible change that should happen in Tokenizer Training from Raw Text.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Local checks

Claim

A concrete answer is on the canvas.

Mechanism

The answer names why the claim should hold.

Bridge

It touches the page context or a neighboring idea.

Misconception check

Grounded research drawerClose

ConceptTokenizer Training from Raw TextAttention & Transformers

Runnable code comparisonTokenizer Training from Raw Text runnable code 1corpus = {"hug": 10, "pug": 5, "pun": 12, "bun": 4, "hugs": 5}Prediction before revealTokenizer Training from Raw Text interactive demoManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes Tokenizer Training from Raw Text click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptAttention & Transformers

Tokenizer Training from Raw Text

Attached question

What is the smallest example that makes Tokenizer Training from Raw Text click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays in this browser, attached to the selected learning item.

Draft noteNext action

No local draft saved.

Evidence to inspect

References to inspect: attached references on this page.
Definition, prerequisite, and contrast concept links
The equation or runnable code that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Tokenizer Training from Raw Text Selected item key: recorded for copy. Context: Attention & Transformers Page anchor: recorded for copy. Open question: What is the smallest example that makes Tokenizer Training from Raw Text click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context

concept/concept-notebook/attention-transformers/tokenizer-training-raw-text
concept:attention-transformers/tokenizer-training-raw-text

Learning Map

Before / Now / Try / Next

BeforeTokenization & Vocabulary Design

NowIntuition -> Math -> Code -> Demo

TryManipulate one control and predict the visible change.

NextRaw Text to Packed Causal-LM Examples (review)

Intuitionready
Mathready
Codeready
Interactive Demoready

AI Companion

Ask beside the selected object

How raw text becomes a learned subword vocabulary: weighted pre-tokens, pair counts, greedy BPE merges, and the merge-order consequences learners can inspect.

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Attention & Transformers concept. Learning surface: Tokenizer Training from Raw Text. What this page says: How raw text becomes a learned subword vocabulary: weighted pre-tokens, pair counts, greedy BPE merges, and the merge-order consequences learners can inspect. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Domain

Attention & Transformers

attention-transformersllm-from-scratchtokenizationdata-pipelinebpe

Prerequisites

Tokenization & Vocabulary Design N-gram Language Models (review)

Leads To

Raw Text to Packed Causal-LM Examples (review)BPE vs. Unigram Tokenization (review)Transformer Implementation From Scratch (planned)

Dataset Cleaning, Filtering, and Deduplication (review)Embedding, Unembedding, and Weight Tying (review)Maximum Likelihood

Within this domain