Transformer systemsGrouped-query attentionPrediction open · result sealed

A living notebook for the math inside modern AI.

What gets smaller when four query heads share one memory?

Make one prediction. Then trace the same change through the head map, equation, code, and calculated memory.

Open the memory lab

About two minutes · saved in this browser only if you lock it in

Attention head sharing

One mechanism, still sealed

Result sealed

Head diagram
Equation
Code
Calculated result

The evaluated count and memory stay sealed until you lock in a guess or choose Just show me.

Start from what is already in front of you

Three useful doors. No blank chat required.

01

I have a mechanism

Test it in the lab

Commit a prediction, inspect the calculation, and keep the boundary of the result visible.

Open the serving lab

02

I have a paper claim

Map the exact object

Turn one claim into the concepts, equations, source checks, and next questions it requires.

Open Paper Mapper

03

I need the foundations

Repair the missing math

Start from intuition, then move through notation, runnable code, and an interactive witness.

Explore foundations

The rule every notebook follows

One idea, carried through four representations.

01IntuitionStart with motion, shape, and analogy.02MathWrite the objects down precisely.03CodeMatch notation to runnable Python.04DemoManipulate the system and watch it respond.

Curated routes

Follow a question, not a catalog.

Open the launch route

01

Foundations First

Build the minimum mathematical language needed to understand optimization and modern model training.

Dot Product Vector Spaces Derivatives Maximum Likelihood

02

Transformer Systems

Move from attention mechanics to the engineering decisions that make long-context inference work.

Dot Product Scaled Dot-Product Attention & Transformer Layers Efficient Attention at Scale: KV Cache, GQA & FlashAttention Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization

03

Frontier Notebooks

Use the foundations to step into generation, alignment, and inspectable model representations.

Scaled Dot-Product Attention & Transformer Layers Sparse Autoencoders: Feature Dictionaries for Mechanistic Interpretability Direct Preference Optimization Diffusion, Score-Based Models & Flow Matching

68 published notebooks

The atlas, kept quiet until you need it.

Good first pagesVector Spaces Derivatives Maximum Likelihood

Linear Algebra9 notebooks Calculus6 notebooks Optimization10 notebooks Machine Learning10 notebooks Probability6 notebooks Information Theory1 notebooks Attention & Transformers12 notebooks Representation Learning3 notebooks Generative Models5 notebooks Scaling6 notebooks Alignment5 notebooks Efficiency5 notebooks LLM Systems6 notebooks Production ML5 notebooks