Domain Neighborhood

Attention & Transformers

The sequence model backbone: tokenization, self-attention, positional encodings, and the transformer block that powers modern LLMs.

7 concepts7 published7 demos

Start with Scaled Dot-Product Attention & Transformer Layers Search Atlas

Selected domain objectScaled Dot-Product Attention & Transformer Layers

Start here. Predict once, then carry the invariant forward.

q_ik_jsoftmaxΣ α_ij v_j

Open first notebook

QuestionWhich invariant should survive into Rotary Position Embeddings (RoPE)?

PredictionBefore the first demo, predict which variable moves first.

Evidence7 demo witnesses in this domain

InvariantName the mechanism before continuing the route.

Learner lensWhat makes this domain feel navigable?

Stabilize the first mechanism, make one prediction, then move one node forward.

Take this lens

Recommended Route

Start here, then follow the prerequisites forward.

This sequence is ordered for learning rather than inventory. Published notebooks with an unavailable prerequisite—or a same-domain route step that depends on one—are labeled in the full inventory instead of being presented as ready steps.

01
Scaled Dot-Product Attention & Transformer Layers
The core transformer operation: compute attention weights from query-key dot products, then mix values to copy information across a sequence.
18 mincodedemoafter Maximum Likelihood, Dot Product
Check Maximum Likelihood first if the symbols feel slippery.
02
Rotary Position Embeddings (RoPE)
A positional encoding that rotates queries and keys so attention depends on relative position via phase differences.
14 mincodedemoafter Scaled Dot-Product Attention & Transformer Layers
Why this follows: Rotary Position Embeddings (RoPE) uses Scaled Dot-Product Attention & Transformer Layers directly.
03
Layer Normalization & RMSNorm
Normalize one token/example vector across features: LayerNorm centers and scales, while RMSNorm keeps RMS-based scaling without mean-centering.
14 mincodedemoafter Scaled Dot-Product Attention & Transformer Layers, Residual Connections & Skip Connections
Why this follows: both pages keep the attention transformers thread active.
04
Tokenization & Vocabulary Design
How text becomes token IDs: segmentation, BPE/unigram tokenizers, and the tradeoffs that shape cost and capability.
14 mincodedemoafter Maximum Likelihood, Representation Learning & Embedding Geometry
Why this follows: both pages keep the attention transformers thread active.
05
Efficient Attention at Scale: KV Cache, GQA & FlashAttention
How attention becomes practical at long context: KV caching for decoding, grouped-query attention, and IO-aware kernels like FlashAttention.
26 mincodedemoafter Scaled Dot-Product Attention & Transformer Layers, Rotary Position Embeddings (RoPE), Efficiency: Quantization, Distillation, LoRA & Sparse MoE
Why this follows: both pages keep the attention transformers thread active.
06
Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization
How frontier LLMs stretch context windows: positional extrapolation (RoPE scaling) plus KV cache memory tricks (GQA, paging, quantization, compression).
22 mincodedemoafter Scaled Dot-Product Attention & Transformer Layers, Rotary Position Embeddings (RoPE), Efficient Attention at Scale: KV Cache, GQA & FlashAttention
Why this follows: Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization uses Efficient Attention at Scale: KV Cache, GQA & FlashAttention directly.
07
FlashAttention: IO-Aware Attention
A fused, tiled attention implementation that avoids materializing the full T x T matrix by using an online softmax, reducing memory traffic and speeding up long-context training/inference.
18 mincodedemoafter Efficient Attention at Scale: KV Cache, GQA & FlashAttention, Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization
Why this follows: FlashAttention: IO-Aware Attention uses Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization directly.

All Published Notebooks

Browse the territory.

Scaled Dot-Product Attention & Transformer Layers

The core transformer operation: compute attention weights from query-key dot products, then mix values to copy information across a sequence.

Level 318 mindemo

Rotary Position Embeddings (RoPE)

A positional encoding that rotates queries and keys so attention depends on relative position via phase differences.

Level 314 mindemo

Layer Normalization & RMSNorm

Normalize one token/example vector across features: LayerNorm centers and scales, while RMSNorm keeps RMS-based scaling without mean-centering.

Level 314 mindemo

Tokenization & Vocabulary Design

How text becomes token IDs: segmentation, BPE/unigram tokenizers, and the tradeoffs that shape cost and capability.

Level 314 mindemo

Efficient Attention at Scale: KV Cache, GQA & FlashAttention

How attention becomes practical at long context: KV caching for decoding, grouped-query attention, and IO-aware kernels like FlashAttention.

Level 426 mindemo

Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization

How frontier LLMs stretch context windows: positional extrapolation (RoPE scaling) plus KV cache memory tricks (GQA, paging, quantization, compression).

Level 422 mindemo

FlashAttention: IO-Aware Attention

A fused, tiled attention implementation that avoids materializing the full T x T matrix by using an online softmax, reducing memory traffic and speeding up long-context training/inference.

Level 418 mindemo

Advanced Bridges

Use these after the core path.

Efficient Attention at Scale: KV Cache, GQA & FlashAttention Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization FlashAttention: IO-Aware Attention