Before opening an advanced page, guess whether what blocks you is notation, optimization, probability, representations, scaling or systems.
Concept atlas
Foundations of Modern Machine Learning
100 concepts behind modern machine learning, from maximum likelihood and attention to generative models, interpretability and alignment. Each page gives a short summary, the key equation and the math behind it, key papers and links to related concepts.
1Core training
2Optimization
3Generative models
4Representations
5Scaling and alignment
6Systems
interactive demospredict, drag, check
Study guide
Start with the map when you are unsure what to learn first.
This atlas100 concepts and the links between them205 prerequisite links · 125 other relations · 40 with interactive demos · 13 study phases
Question
Which simpler idea do I need first to understand the method I am stuck on?
How to use this mapPredict · Check · Generalize · Continue
The map shows which concepts build on which; each page cites its key papers; 40 pages include an interactive demo.
The most useful concepts explain several models, papers or engineering trade-offs at once.
Browse the map for neighboring concepts, follow the study order when sequence matters, or filter the list below to pages with demos.
Recommended Study Order
A suggested reading order, from fundamentals to recent research. Most concepts come after all of their prerequisites; some depend on a concept listed later.
1
Core probabilistic training and transformers
2
Architecture fundamentals
3
Optimization & generalization
4
Generative modeling families
5
Representation & interpretability
6
Modern efficiency & inference
7
Alignment & RLHF
8
Scaling, theory & multimodal
9
Advanced architectures & generation
10
Mathematical foundations & information geometry
11
Reasoning at inference time & synthetic data
12
Advanced alignment & safety research
13
Recent research (2024–2025)
All Concepts
Search by concept name, equation, paper title or author.
1
Maximum Likelihood
Maximum Likelihood, Cross-Entropy & KL Divergence
Open the reference →2
Attention
Scaled Dot-Product Attention & Transformer Layers
Open the reference →3
Adam
Adam & Adaptive Gradient Methods
Open the reference →4
Sharpness
Loss Landscapes, Sharpness & Flat Minima
Open the reference →5
Double Descent
Overparameterization & Generalization, Double Descent
Open the reference →6
NTK
Neural Tangent Kernel & Infinite-Width Limits
Open the reference →7
VAEs
Variational Autoencoders & Variational Inference
Open the reference →8
GANs
GANs & Adversarial Divergence Minimization
Open the reference →9
Diffusion
Diffusion, Score-Based Models & Flow Matching
Open the reference →10
Embeddings
Representation Learning & Embedding Geometry
Open the reference →11
Superposition
Superposition, Sparse Features & Monosemanticity
Open the reference →12
Probing
Probing, Linear Classifier Probes & Activation Analysis
Open the reference →13
Circuits
Transformer Circuits, Induction Heads & Mechanistic Interpretability
Open the reference →14
Scaling Laws
Scaling Laws & Emergent Abilities
Open the reference →15
RLHF
Preference-Based Alignment: RLHF, Reward Modeling, Constitutional AI
Open the reference →16
Efficiency
Efficiency: Quantization, Distillation, LoRA & Sparse MoE
Open the reference →17
Learning Theory
Theoretical Foundations: PAC Learning, MDL & Information Bottleneck
Open the reference →18
RoPE
Rotary Position Embeddings (RoPE)
Open the reference →19
Efficient Attention
Efficient Attention at Scale: KV Cache, GQA & FlashAttention
Open the reference →20
Speculative Decoding
Speculative Decoding: Lossless Multi-Token Generation
Open the reference →21
LLM Serving
LLM Serving at Scale: Prefill, Decode & Continuous Batching
Open the reference →22
MoE
Sparse Mixture of Experts: Routing, Load Balancing & Expert Parallelism
Open the reference →23
MoE Serving
MoE Serving & Scheduling: Token Dispatch, All-to-All, Disaggregated Parallelism
Open the reference →24
DPO
Direct Preference Optimization: RL-Free Alignment from Human Preferences
Open the reference →25
KTO
KTO: Alignment from Binary Feedback via Human-Aware Losses
Open the reference →26
Reward Hacking
Reward Hacking & Overoptimization: Goodhart's Law in Preference Optimization
Open the reference →27
Sparse Autoencoders
Sparse Autoencoders at Scale: Feature Dictionaries for Mechanistic Interpretability
Open the reference →28
Circuit Discovery
Automated Circuit Discovery: Patching, Attribution & Decomposition at Scale
Open the reference →29
Activation Steering
Activation Steering: Feature-Guided Interventions for Inference-Time Control
Open the reference →30
Long Context
Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization
Open the reference →31
State Space Models
State Space Models & Hybrid Architectures: Mamba-2, Jamba, Griffin
Open the reference →32
Multimodal Pretraining
Multimodal Foundations: Vision Encoders, Contrastive Learning & Cross-Attention Fusion
Open the reference →33
Tokenization
Tokenization & Vocabulary Design
Open the reference →34
Decoding & Sampling
Decoding & Sampling: Temperature, Top-p & Inference-Time Control
Open the reference →35
Backpropagation
Backpropagation & Automatic Differentiation
Open the reference →36
Score Matching
Score Matching & Score-Based Generative Models
Open the reference →37
In-Context Learning
In-Context Learning: Learning Without Weight Updates
Open the reference →38
Optimal Transport
Optimal Transport & Wasserstein Distance
Open the reference →39
Normalizing Flows
Normalizing Flows: Exact Likelihood via Invertible Transforms
Open the reference →40
PPO
PPO: Proximal Policy Optimization
Open the reference →41
Residual Connections
Residual Connections & Skip Connections
Open the reference →42
Classifier-Free Guidance
Classifier-Free Guidance in Diffusion
Open the reference →43
RAG
Retrieval-Augmented Generation (RAG)
Open the reference →44
Adversarial Examples
Adversarial Examples & Robustness
Open the reference →45
Grokking
Grokking: Delayed Generalization
Open the reference →46
Logit Lens
Logit Lens: Probing Intermediate Representations
Open the reference →47
Learning-Rate Schedules
Learning Rate Schedules: Warmup, Decay & Cycling
Open the reference →48
Weight Initialization
Weight Initialization: Xavier, He & µP
Open the reference →49
Contrastive Learning
Contrastive Learning & InfoNCE
Open the reference →50
Distributed Training
Distributed Training: Data, Tensor & Pipeline Parallelism
Open the reference →51
Beam Search
Beam Search & Structured Decoding
Open the reference →52
Dropout
Dropout: Stochastic Regularization
Open the reference →53
Energy-Based Models
Energy-Based Models & Score Functions
Open the reference →54
Layer Normalization
Layer Normalization & RMSNorm
Open the reference →55
Fisher Information
Fisher Information & Information Geometry
Open the reference →56
Natural Gradient
Natural Gradient & Riemannian Optimization
Open the reference →57
SGD & Momentum
Stochastic Gradient Descent & Momentum
Open the reference →58
Weight Decay & AdamW
Weight Decay & AdamW: Decoupled Regularization
Open the reference →59
Gradient Clipping
Gradient Clipping & Explosion Prevention
Open the reference →60
Label Smoothing
Label Smoothing & Soft Targets
Open the reference →61
Batch Normalization
Batch Normalization
Open the reference →62
Knowledge Distillation
Knowledge Distillation: Learning from Teachers
Open the reference →63
Quantization
Quantization: Compressing Models to Integers
Open the reference →64
Pruning
Pruning: Removing Unnecessary Weights
Open the reference →65
Self-Supervised Learning
Self-Supervised Learning: Labels from Structure
Open the reference →66
Calibration
Calibration & Temperature Scaling
Open the reference →67
SwiGLU
SwiGLU & Gated Activations
Open the reference →68
FlashAttention
FlashAttention: IO-Aware Attention
Open the reference →69
Constitutional AI
Constitutional AI: Principles-Based Alignment
Open the reference →70
Bregman Divergence
Bregman Divergence & Mirror Descent
Open the reference →71
Reproducing Kernels
Reproducing Kernel Hilbert Spaces
Open the reference →72
Persistent Homology
Persistent Homology & Topological Data Analysis
Open the reference →73
Lie Groups
Lie Groups & Equivariant Networks
Open the reference →74
Test-Time Compute
Test-Time Compute & Inference Scaling
Open the reference →75
Chain of Thought
Chain-of-Thought Prompting
Open the reference →76
World Models
World Models & Model-Based RL
Open the reference →77
Synthetic Data
Synthetic Training Data
Open the reference →78
Consistency Models
Consistency Models
Open the reference →79
Activation Checkpointing
Activation Checkpointing & Memory Efficiency
Open the reference →80
GQA
Grouped-Query Attention (GQA)
Open the reference →81
Process Reward Models
Process Reward Models
Open the reference →82
RLAIF
RLAIF: Reinforcement Learning from AI Feedback
Open the reference →83
Flow Matching
Flow Matching & Rectified Flows
Open the reference →84
Instruction Tuning
Instruction Tuning
Open the reference →85
Deliberative Alignment
Deliberative Alignment
Open the reference →86
Debate
AI Safety via Debate
Open the reference →87
Iterated Amplification
Iterated Amplification
Open the reference →88
Weak-to-Strong
Weak-to-Strong Generalization
Open the reference →89
Automated Red Teaming
Automated Red Teaming
Open the reference →90
Mesa-Optimization
Mesa-Optimization & Inner Alignment
Open the reference →91
Sleeper Agents
Sleeper Agents & Alignment Faking
Open the reference →92
LLM-as-Judge
Model-Graded Evaluations
Open the reference →93
Capability Elicitation
Capability Elicitation & Eliciting Latent Knowledge
Open the reference →94
Sandwiching Evaluations
Sandwiching Evaluations
Open the reference →95
Mixture-of-Depths
Mixture-of-Depths
Open the reference →96
Tree Search
Tree Search for Language-Model Reasoning
Open the reference →97
Video World Models
Video World Models
Open the reference →98
Self-Improvement
Self-Improvement & Distillation Loops
Open the reference →99
Model Collapse
Model Collapse
Open the reference →100
Infinite Context
Infinite-Context Architectures
Open the reference →About These Pages
What each page contains
A short summary, questions or points that are easy to miss, the key equation and the math behind it, key papers and links to related concepts.
Interactive demos
40 of the 100 pages include an interactive demo. Use the Demos filter above to list them.
Newer notebooks
Some concepts also have a newer notebook with its own demo. Those pages link to it at the top.