Transformer Lab
KV memory to serving pressure
Start from one exact equation, make the MHA-to-GQA prediction, reveal the memory evidence, then carry the invariant into long context and serving.
Mem_KV = B * N_layers * T * H_kv * d_head * 2 * bytesA single memory equation carries the route from attention mechanics into long-context and serving pressure.
BN_layersTH_kvd_headbytesTransformer Lab Route
KV memory to long-context pressure
Checking local routeReading browser memory before opening saved evidence
- 01Text to one update
- 02Tokens and position
- 03Attention routing
- 04RoPE phase
- 05KV memory
- 06Long-context pressure
- 07Serving and decoding
- 08Speculative decoding
- 09Evaluation and falsification
- 10Capstone systems claim
Operable Station 00
From text to one gradient update
Start with a toy text object, choose a tokenization, route the final context token through one causal attention row, then inspect the exact softmax-cross-entropy update that follows.
CF.LABCASE.V1Prediction neededNo prediction yet is preserved as the first answer.
Mechanism sourceAttention Is All You NeedQueries, keys, values, causal paths, and next-token loss.Tokenizer sourceSentencePieceUse only as motivation: this station uses a toy subword vocabulary.Loss sourceCS231n SoftmaxScores become probabilities before cross-entropy.Code witnessD2L Softmax RegressionLinear logits, softmax, cross-entropy, SGD.
Selected Object
Toy next-token training case
the model learns patternsThe tokenizer here is deliberately small and inspectable. It is not BPE, unigram LM, or real SentencePiece; it is a controlled witness for how token boundaries change positions before the loss. For the final training row, the context is everything before the last token and y is the final token.
0the1model2learns3patterns
4 toy tokens3 next-token positionsh=[0.83, 0.76, 0.31]q position 2
Prediction Checkpoint
After scaling the model key by 2x, which source token receives the largest gradient signal from the final loss?
First answerNo prediction yetRevisions never overwrite the first answer.
Manipulation
Change one axis, then re-predict.
toy tokenizer
target y
learning rate eta
model key scale
attention temperature
causal mask
residual path
Fewer pieces, fewer training positions.
Operable Station 01
One token, one attention-gradient step
Work on the smallest training step that still has a real attention route: one final-token query, two keys, two values, one target, and a scalar loss.
cf.labCase.v1Preparing local casePrediction and controls persist in this browser.
- 01Datasource-mapped
- 02Tokenizationsource-mapped
- 03Causal attentionoperable
- 04Gradient traceactive
- 05RoPEmapped
- 06KV/GQAready
- 07Efficient exact attentionmapped
- 08Long contextmapped
- 09Stable pretrainingpreview
- 10Post-trainingpreview
- 11Evaluationpreview
- 12Serving claimready
Selected Object
Final-token value-gradient creditd_k = 1; dL/dv_j = (y_hat - y) * alpha_jThis station tests whether value updates follow raw values, target distance, or the attention weights that routed each value into the output.
Prerequisite Gap
Values are not attention weights.The raw value can be large while its gradient is small. Credit flows through the softmax row that used the value.
Transfer
Change keys, then ask again.The next station should make the key path visible: changing k_0/k_1 changes alpha, which changes the value gradients.
First Prediction
Which value scalar receives the larger gradient magnitude?
First answerNo prediction yetRevisions never overwrite the first answer.
Manipulation
Move one axis, then re-predict.
q_1 and key contrast reroute attention. Values and target change shared error. Learning rate changes the update preview, not the gradient assignment.
q_1: 1.00
k_1 - k_0: 0.60
v_0: 0.20
v_1: 1.20
target y: 0.85
learning rate: 0.30
ObjectShapeCurrent value
d_k[1]
fixed scalar sliceq_1[1]
1.000k[2]
[-0.300, 0.300]m[2]
[0, 0] final-token rowv[2]
[0.200, 1.200]alpha[2]
hidden until revealy_hat[1]
hidden until revealL[1]
hidden until revealtoken 0token 1
score_0-0.300
score_10.300
alpha_0hidden
alpha_1hidden
Forward Trace
s = q_1 * kalpha = softmax(s)y_hat = alpha_0 * v_0 + alpha_1 * v_1L = 0.5 * (y_hat - y)^2
Operable Station 03
RoPE: carry the query-key route into phase
Keep the same query/key attention object from the route. Now ask which part of the score survives when position changes.
CF.LABCASE.V1Prediction neededNo prediction yet is preserved as the first answer.
Mechanism sourceRoFormer / RoPERotary position embedding as relative phase in Q/K pairs.Route sourceAttention Is All You NeedRoPE changes the query-key score geometry, not the value-copy rule.
Selected Object
One 2D query/key frequency pair
(R_p q)^T (R_q k) = q^T R(theta_q - theta_p) kUse one tiny coordinate pair so the cancellation is inspectable. The content vectors stay fixed; only the positions and frequency band move.
q[0.72, 0.32]k[0.58, 0.45]omega0.42
Open RoPE notebook objectPrediction Checkpoint
Shift both positions by +3: q 2 -> 5, k 0 -> 3. What should happen to the query-key score?
First answerNo prediction yetRevisions never overwrite the first answer.
Manipulation
Move positions, then re-predict.
query position p
key position q
equal shift
frequency band
Default witness band: enough movement to see the counterfactual.
Want the context first? See the source notebook. The result stays hidden here.