Domain
Attention & Transformers
The transformer architecture: tokenization, self-attention, positional encodings and transformer blocks, with calculations of attention and its memory cost at long context.
Preview the recommended route
Start here. Make one prediction, then note what carries over to the next notebook.
q_ik_jsoftmaxΣ α_ij v_jLearnerWhich notebook should I read first?
Open the first notebookRead the first notebook, make one prediction in its demo, then go on to the next.
Recommended route
Start here, then follow the prerequisites forward.
Read these in order: each notebook comes after the notebooks in this domain that it builds on, and the label beside it gives its level. A prerequisite with no page here yet is marked as outside reading. A notebook is left out of the route only when a prerequisite has neither a page here nor a named outside reading; the full list below marks it.
- 01Scaled Dot-Product Attention & Transformer Layers
How a query scores keys with scaled dot products, turns the scores into softmax weights and mixes the values; with masks, several queries at once and the transformer layer around them.
Intermediate18 mincodedemoafter Maximum Likelihood, Dot Product - 02Positional Encoding
How transformers represent order: sinusoidal encodings, learned embeddings, and relative-position methods like RoPE.
Introductory12 mincodedemoafter Scaled Dot-Product Attention & Transformer LayersPositional Encoding builds directly on Scaled Dot-Product Attention & Transformer Layers.
- 03Residual Connections & Skip Connections
A residual connection adds a layer's output to its input, giving the input a direct path through the network while the layer learns an update.
Intermediate14 mincodedemoafter Scaled Dot-Product Attention & Transformer Layers - 04Rotary Position Embeddings (RoPE)
A positional encoding that rotates queries and keys so attention depends on relative position via phase differences.
Intermediate14 mincodedemoafter Scaled Dot-Product Attention & Transformer Layers - 05Layer Normalization & RMSNorm
Normalizing each token's vector across its features: LayerNorm subtracts the mean and divides by the standard deviation; RMSNorm only divides by the root mean square.
Intermediate24 mincodedemoafter Scaled Dot-Product Attention & Transformer Layers, Residual Connections & Skip Connections - 06Tokenization & Vocabulary Design
How text becomes token IDs: segmentation, BPE/unigram tokenizers, and the tradeoffs that shape cost and capability.
Intermediate14 mincodedemoafter Maximum Likelihood, Representation Learning & Embedding Geometry - 07SwiGLU: Gated MLP Blocks in Transformers
Why modern transformer MLPs often use a learned multiplicative gate: one projection proposes a token-local write while another projection controls how much of each channel reaches the residual stream.
Advanced16 mincodedemoafter Scaled Dot-Product Attention & Transformer Layers, Linear Transformations (outside reading) - 08Efficient Attention at Scale: KV Cache, GQA & FlashAttention
How attention becomes practical at long context: KV caching for decoding, grouped-query attention, and IO-aware kernels like FlashAttention.
Advanced26 mincodedemoafter Scaled Dot-Product Attention & Transformer Layers, Rotary Position Embeddings (RoPE), Efficiency: Quantization, Distillation, LoRA & Sparse MoE - 09Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization
How models handle long inputs: keeping positions within the range seen in training (RoPE scaling), sliding windows, and cutting KV-cache memory with GQA, paging, quantization and compression.
Advanced22 mincodedemoafter Scaled Dot-Product Attention & Transformer Layers, Rotary Position Embeddings (RoPE), Efficient Attention at Scale: KV Cache, GQA & FlashAttentionLong Context Engineering builds directly on Efficient Attention at Scale.
- 10FlashAttention: IO-Aware Attention
Exact attention computed tile by tile with a running softmax, so the full T × T matrix is never written to GPU main memory; in the paper's benchmarks the reduced memory traffic made attention faster.
Advanced30 mincodedemoafter Efficient Attention at Scale: KV Cache, GQA & FlashAttention, Long Context Engineering: RoPE Scaling, KV Compression & Memory OptimizationFlashAttention builds directly on Long Context Engineering.
All notebooks
Every published notebook in this domain.
Scaled Dot-Product Attention & Transformer Layers
How a query scores keys with scaled dot products, turns the scores into softmax weights and mixes the values; with masks, several queries at once and the transformer layer around them.
Positional Encoding
How transformers represent order: sinusoidal encodings, learned embeddings, and relative-position methods like RoPE.
Residual Connections & Skip Connections
A residual connection adds a layer's output to its input, giving the input a direct path through the network while the layer learns an update.
Rotary Position Embeddings (RoPE)
A positional encoding that rotates queries and keys so attention depends on relative position via phase differences.
Layer Normalization & RMSNorm
Normalizing each token's vector across its features: LayerNorm subtracts the mean and divides by the standard deviation; RMSNorm only divides by the root mean square.
Tokenization & Vocabulary Design
How text becomes token IDs: segmentation, BPE/unigram tokenizers, and the tradeoffs that shape cost and capability.
SwiGLU: Gated MLP Blocks in Transformers
Why modern transformer MLPs often use a learned multiplicative gate: one projection proposes a token-local write while another projection controls how much of each channel reaches the residual stream.
Efficient Attention at Scale: KV Cache, GQA & FlashAttention
How attention becomes practical at long context: KV caching for decoding, grouped-query attention, and IO-aware kernels like FlashAttention.
Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization
How models handle long inputs: keeping positions within the range seen in training (RoPE scaling), sliding windows, and cutting KV-cache memory with GQA, paging, quantization and compression.
FlashAttention: IO-Aware Attention
Exact attention computed tile by tile with a running softmax, so the full T × T matrix is never written to GPU main memory; in the paper's benchmarks the reduced memory traffic made attention faster.