toy text and tokenization
Transformer Systems Lab
From text to one gradient update
Begin with a tiny object instead of a course grid: a string, two toy tokenizations, one causal attention row, a cross-entropy loss, and one inspectable update.
Selected objectthe model learns patterns
tokens -> QK route -> softmax CE -> gradienttokens, attention paths, p-y
commit source token
tokenizer, key scale, mask
trace, table, finite check
path controls gradient
residual-off perturbation
RoPE phase geometry
Launch standardEvidence opens only after a prediction or an explicit skip in later stations.
Operability contract
The page is complete only when the loop closes around the object.
Operable Station 00
From text to one gradient update
Start with a toy text object, choose a tokenization, route the final context token through one causal attention row, then inspect the exact softmax-cross-entropy update that follows.
CF.LABCASE.V1Prediction neededNo prediction yet is preserved as the first answer.
Mechanism sourceAttention Is All You NeedQueries, keys, values, causal paths, and next-token loss.Tokenizer sourceSentencePieceUse only as motivation: this station uses a toy subword vocabulary.Loss sourceCS231n SoftmaxScores become probabilities before cross-entropy.Code witnessD2L Softmax RegressionLinear logits, softmax, cross-entropy, SGD.
Selected Object
Toy next-token training case
the model learns patternsThe tokenizer here is deliberately small and inspectable. It is not BPE, unigram LM, or real SentencePiece; it is a controlled witness for how token boundaries change positions before the loss. For the final training row, the context is everything before the last token and y is the final token.
0the1model2learns3patterns
4 toy tokens3 next-token positionsh=[0.83, 0.76, 0.31]q position 2
Prediction Checkpoint
After scaling the model key by 2x, which source token receives the largest gradient signal from the final loss?
First answerNo prediction yetRevisions never overwrite the first answer.
Manipulation
Change one axis, then re-predict.
toy tokenizer
target y
learning rate eta
model key scale
attention temperature
causal mask
residual path
Fewer pieces, fewer training positions.
Transformer Systems Lab route
One complete station first; the whole route stays visible.
Later stations must inherit the same object-centered contract: object, gap, prediction, manipulation, evidence, invariant, transfer, and one next move.
- 01Data, tokenization, objective
Toy text -> tokens -> next-token loss
operable - 02Causal self-attention
One row of QK^T, mask, softmax, V
operable in cockpit - 03Reverse-mode gradient trace
Loss signal through attention and residual paths
operable in cockpit - 04RoPE
Rotated query/key pair and phase gap
first-class next - 05KV cache and grouped-query attention
Mem_KV = B * N_layers * T * H_kv * d_head * 2 * bytes
operable in cockpit - 06Efficient exact attention
Exact attention with different memory movement
source mapped - 07Long-context use
Available context vs usable retrieval
first-class station - 08Stable pretraining
Activation scale, update ratio, loss trace
planned - 09Post-training
Reference policy, preference pair, behavior shift
planned - 10Evaluation and falsification
Claim, slice, metric, counterexample
first-class station - 11Serving and deployment constraints
Workload, prefill, decode, cache pressure
first-class station - 12Capstone systems claim
One defensible architecture/training/eval/serving claim
first-class station