The Lab

Everything here asks you to commit to an answer first.

Investigations rather than lessons. Each one puts a mechanism in front of you, takes your prediction, and only then shows the calculation — because a prediction you have already made is the only thing that makes the result informative.

Stations: 8, in sequence
State: Saved in this browser only
Account: Not required, not offered
Scoring: None

What an investigation looks like

Before the list, one worked example. Everything below is live: the numbers inside the sentences are controls, so drag them and read what changes. Nothing is saved, and there is no answer to get right.

A transformer serving one sequence keeps a key and a value for every token it has already read, at every layer. Take a model of with , serving in . If every query head keeps its own key and value, that cache is 4.0 GiB — for one user.

Now share one key/value pair across . The model still runs 32 query heads, but it now stores only 8 key/value heads per layer, and the cache falls to 1.0 GiB — 75% less.

Query heads per KV head4:1

8 key/value heads per layer.

Layers32

Precisionfp16

2 bytes per stored value.

One KV head per query headGrouped, 4:1One KV head for all queries

Cache size grows linearly with context, and the slope is set entirely by how many key/value heads you keep. Sharing changes the slope, not the shape — which is why it buys proportionally more the longer the context gets.

bytes = 2 × layers × kv_heads × head_dim × tokens × bytes_per_value

  head_dim  = d_model / query_heads     (4096 / 32 = 128)
  kv_heads  = query_heads / group_size  (32 / 4 = 8)
  the 2     = one key and one value per token

The arithmetic, so the numbers above can be checked rather than trusted.

Transformer systems

One track, worked in order. It starts at a single gradient update and ends with you making a defensible claim about a serving tradeoff. Each station is self-contained enough to open on its own, but the sequence is the argument.

From text to one gradient updateStation 01Follow a single batch from tokens to a weight change, with every intermediate quantity visible.Grouped-query attention and what the model storesStation 02Predict the memory effect of sharing one KV head across four query heads, then inspect the calculation.RoPE phaseStation 03Position as rotation: where the phase comes from and what breaks when you extend it.Long contextStation 04What actually grows with sequence length, and which of those growths you can trade away.Serving and decodingStation 05Batching, cache pressure, and the point where a latency budget starts choosing the architecture.Speculative decodingStation 06Why a smaller draft model can make a larger one faster without changing what it outputs.Evaluation and falsificationStation 07How to design a measurement that could have come out against you — the station Atlas-64 needed.Capstone: make a systems claimStation 08State a claim about a serving tradeoff, bound it, and show what evidence would settle it.

Guided paths

Longer routes that thread the atlas and the stations together, for when you want the whole territory rather than one mechanism.

Attention to serving, end to endGuided pathThe long route: mechanism, then cost, then the decisions that cost forces.KV memory to serving pressureGuided pathA shorter arc through the same territory, anchored on one quantity.All learning pathsIndexEvery curated route through the atlas.

Single-sitting checks

Small, complete, and stateless. Useful when you have ten minutes and want to find out whether you actually understand something you believe you understand.

Today's mechanism checkOne sittingA single prediction against a single mechanism. No score, no streak, no account.The 4× KV-cache challengeOne sittingA question first, an equation second. Commit to an answer before the arithmetic appears.