The Atlas

Find the idea you need.

Search a concept or question. New to attention? Start with one guided calculation, then explore the full register.

Search the Atlas

Search published concepts, foundations, reference notes, and source links.

New to attention? Start here

How do comparisons become a mixture?

Follow one query through attention. Aim to explain how scores become weights, and how those weights carry information from values into an output.

Open the guided attention notebook

One query · three sources · an unmasked worked example. No prediction gate.

Follow the calculation: vectors → compare → normalize → mix

Computational handoffs, not prerequisite edges. This is an authored reading route through existing pages. The arrows say what the calculation passes forward; they do not change the concept prerequisite graph.

Vectors
Query, keys, values
Give the query and each key matching coordinates. Keep the values as the information to carry.
Read Vector Spaces
Query + keys → comparisons
Compare
Scaled scores
Multiply matching query–key coordinates, add, then divide by the square root of the key dimension.
Read Dot Product
Scores → normalization
Normalize
Attention weights
Softmax turns the score row into nonnegative weights whose sum is one.
Inspect Weights in the notebook
Weights + values → mixture
Mix
Output vector
Multiply each value by its weight, then add those contributions. The output lives in value space.
Inspect Mixture in the notebook
One weighted output

Normalize and Mix open the same notebook. Select Weights or Mixture there; these are views of one calculation, not separate lessons. Query and key vectors share a dimension; values may have a different dimension.

Need a foundation? Repair just the missing idea

These are optional reading repairs, not a required course or a readiness score. The concept pages keep their own prerequisite lists.

What can I add and scale?Vector Spaces
How do two vectors become a score?Dot Product
What does a total weight of one mean?Probability Basics

The notebook also offers Practice and transfer and Advanced demonstrations. Choose those activities inside the concept; opening a route here does not record progress or establish mastery.

See all learning routes, including systems and serving

Full register

51 published notes · 12 topics · 100 foundations · 4 standalone notes and references · 54 source links

Linear Algebra

Vectors, matrices, and linear maps: the language of representations, optimization, and modern deep learning. Open the topic.

Dot Productfigure · code · 12 minThe dot product measures alignment: it connects angles, lengths, and projections, and underlies cosine similarity in ML.Vector Spacesfigure · code · 10 minA vector space is a set of objects you can add and scale, where those operations behave consistently.

Calculus

Rates of change and accumulation. Calculus is the language behind gradients, optimization, continuous-time dynamics, and why backprop works as efficiently as it does. Open the topic.

Backpropagationfigure · code · 15 minBackpropagation applies reverse-mode autodiff to neural networks so one scalar loss can train many parameters.Computation Graphsfigure · code · 13 minA computation graph breaks a calculation into nodes so values flow forward and sensitivities flow backward.Derivativesfigure · code · 14 minThe derivative is an instantaneous rate of change: the slope you get when a secant line becomes a tangent line.Reverse-Mode Automatic Differentiationfigure · code · 14 minReverse-mode autodiff computes gradients by sending cotangents backward through a computation graph.

Optimization

How we train models: gradients, learning rates, curvature, and the practical tricks that make deep nets converge. Open the topic.

Adam Optimizerfigure · code · 16 minAdam is an adaptive optimizer that combines momentum (EMA of gradients) with per-parameter RMS normalization (EMA of squared gradients).Gradient Descentfigure · code · 12 minGradient descent turns local slope information into an iterative update rule for reducing a loss.Learning Rate Schedules: Warmup, Decay & Cyclingfigure · code · 14 minSchedule shapes that change the scalar learning-rate scale over training, with sourced CLR/range-test and SGDR cosine-restart examples plus caveated warmup/decay teaching patterns.Loss Landscapes, Sharpness & Flat Minimafigure · code · 16 minHow 2D loss slices, Hessian curvature, SAM-style neighborhood loss, and a toy 2/eta stability line expose local sensitivity during optimization.

Probability

Uncertainty made precise: events, random variables, expectations, and the distributions that models learn. Open the topic.

Bayesian Inferencefigure · code · 17 minBayesian inference updates a prior distribution over unknowns into a posterior by multiplying by the likelihood and normalizing.Cross-Entropyfigure · code · 16 minCross-entropy is the target-weighted surprise of a model distribution; in deep learning it is the bridge from likelihood to a differentiable training loss.Distributionsfigure · code · 15 minA distribution is the law of a random variable: it says how probability mass or density lands on the values the variable can take.Maximum Likelihoodfigure · code · 18 minMaximum likelihood fits parameters by making the observed data most probable; for classifiers it becomes negative log-likelihood, cross-entropy, and a KL fit to the empirical distribution.Probability Basicsfigure · code · 12 minEvents are subsets of a sample space, and probabilities obey a few axioms; from there you get conditional probability, independence, and Bayes' rule.Random Variablesfigure · code · 14 minA random variable is a function from outcomes to numbers; its distribution lets you compute expectations, variances, and likelihoods.

Information Theory

How we measure information and mismatch between distributions: entropy, cross-entropy, KL divergence, mutual information, and why they appear everywhere in ML. Open the topic.

KL Divergence (Relative Entropy)figure · code · 14 minKL divergence is a directional expected log-probability mismatch between distributions; it explains cross-entropy training, variational inference, and KL-regularized alignment.

Attention & Transformers

The sequence model backbone: tokenization, self-attention, positional encodings, and the transformer block that powers modern LLMs. Open the topic.

Efficient Attention at Scale: KV Cache, GQA & FlashAttentionfigure · code · 26 minHow attention becomes practical at long context: KV caching for decoding, grouped-query attention, and IO-aware kernels like FlashAttention.FlashAttention: IO-Aware Attentionfigure · code · 18 minA fused, tiled attention implementation that avoids materializing the full T x T matrix by using an online softmax, reducing memory traffic and speeding up long-context training/inference.Layer Normalization & RMSNormfigure · code · 14 minNormalize one token/example vector across features: LayerNorm centers and scales, while RMSNorm keeps RMS-based scaling without mean-centering.Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimizationfigure · code · 22 minHow frontier LLMs stretch context windows: positional extrapolation (RoPE scaling) plus KV cache memory tricks (GQA, paging, quantization, compression).Rotary Position Embeddings (RoPE)figure · code · 14 minA positional encoding that rotates queries and keys so attention depends on relative position via phase differences.Scaled Dot-Product Attention & Transformer Layersfigure · code · 18 minFollow one query, several rows, explicit source permissions and diagnostic cases through the same attention mechanism.Tokenization & Vocabulary Designfigure · code · 14 minHow text becomes token IDs: segmentation, BPE/unigram tokenizers, and the tradeoffs that shape cost and capability.

Representation Learning

Embeddings and the geometry of meaning: similarity, normalization, contrastive objectives, and why vector spaces become usable interfaces for models. Open the topic.

Representation Learning & Embedding Geometryfigure · code · 18 minHow models turn inputs into vectors whose geometry can expose useful factors, contextual meaning, and similarity structure.Sparse Autoencoders: Feature Dictionaries for Mechanistic Interpretabilityfigure · code · 16 minSparse autoencoders learn a reusable dictionary of feature directions so dense model activations can be explained by a small set of interpretable latent factors.

Generative Models

How models generate: likelihood, latent variables, diffusion/score models, flows, and the training tricks that make sampling work. Open the topic.

Diffusion, Score-Based Models & Flow Matchingfigure · code · 22 minDenoise noise into data: the diffusion forward process, score matching, and modern sampling via reverse-time dynamics and flow matching.Flow Matching & Rectified Flowsfigure · code · 16 minLearn flow matching as velocity-label regression: straight conditional paths use x_t=(1-t)x0+t x1 but supervise the full target u_t=x1-x0.Normalizing Flows: Tractable Density via Invertible Transformsfigure · code · 18 minInvertible transforms trained with change-of-variables: tractable transformed densities and sampling, with architectural constraints to make the needed Jacobian determinants tractable.Score Matching & Score-Based Generative Modelsfigure · code · 18 minLearn the score field grad_x log p(x) without normalizing constants. Denoising score matching turns diffusion training into simple regression on noise.Variational Autoencodersfigure · code · 20 minA latent-variable model trained by maximizing an evidence lower bound; the gap is KL(q_phi(z|x) || p_theta(z|x)), so the encoder is learned inference rather than just compression.

Scaling

How loss and capability change with parameters, data, and compute; how to allocate a training budget; and why some abilities appear suddenly at scale. Open the topic.

Neural Tangent Kernel (NTK) & Infinite-Width Limitsfigure · code · 18 minA limit where wide neural networks behave like kernel methods: the model linearizes around initialization and training becomes kernel regression.Overparameterization & Generalization (Double Descent)figure · code · 16 minTest error can peak at the interpolation threshold then fall again as models get larger: why modern overparameterized nets still generalize.Scaling Laws & Emergent Abilitiesfigure · code · 18 minEmpirical power laws that predict how loss and capability improve with parameters, data, and compute, and how to choose compute-optimal training runs.Test-Time Compute: Spending Inference Budget on Searchfigure · code · 22 minTest-time compute spends extra inference budget on sampling, verification, and selection; it helps when the generator can produce good candidates and the verifier ranks them reliably.Tree Search Reasoning: Allocating Inference Budget Across Prefixesfigure · code · 24 minTree search spends inference budget on partial reasoning prefixes, using local verifier scores, frontier expansion, and max backups to decide which branches deserve more thought.

Alignment

How we shape model behavior: preference learning, reward modeling, KL-regularized fine-tuning, and the failure modes that appear when you optimize the wrong thing. Open the topic.

Direct Preference Optimizationfigure · code · 18 minDPO turns pairwise preferences into binary cross-entropy on reference-relative log odds, using the KL-regularized RLHF optimum to make the policy itself an implicit reward model.Kahneman-Tversky Optimizationfigure · code · 18 minKTO turns binary desirable/undesirable labels into a reference-relative utility loss: push a labeled output's policy/reference log-ratio above or below a KL-derived baseline, with saturating gradients.Process Reward Models: Step-Level Verifiers for Reasoningfigure · code · 24 minA process reward model scores intermediate reasoning steps instead of only terminal answers, giving denser verifier feedback for reranking and search while remaining a learned proxy.Reward Hacking: Overoptimizing Preference Proxiesfigure · code · 22 minWhen an imperfect preference proxy is optimized past its validation regime, policy mass shifts toward reward-model errors; KL, ensembles, LCBs, and monitoring slow this down but do not make the proxy true.RLHF: Reward Modeling + KL-Regularized Policy Optimizationfigure · code · 20 minRLHF trains a reward model from pairwise preferences, then reweights a reference policy toward high learned reward while a KL penalty limits distribution shift.

Efficiency

How we make models cheaper to train and serve: quantization, distillation, low-rank adapters, sparsity, and the memory/latency tradeoffs that dominate real deployments. Open the topic.

Efficiency: Quantization, Distillation, LoRA & Sparse MoEfigure · code · 20 minThe practical toolkit for making big models cheaper: quantize weights/activations, distill teachers into students, adapt with low-rank updates (LoRA), and use sparsity (MoE).Knowledge Distillation: Learning from Teachersfigure · code · 16 minTrain a smaller student to mimic a stronger teacher by matching soft probability distributions (often with temperature), transferring 'dark knowledge' beyond hard labels.Pruning: Removing Unnecessary Weightsfigure · code · 16 minReduce parameter count by zeroing or removing weights. Unstructured sparsity needs sparse kernels for speed; structured pruning removes whole channels/heads to shrink dense tensor shapes.Quantization: Compressing Models to Integersfigure · code · 16 minReduce memory and bandwidth by storing weights/activations in low-bit integers (INT8/INT4) with careful scaling to limit accuracy loss.Sparse Mixture of Experts: Routing, Load Balancing & Expert Parallelismfigure · code · 20 minConditional computation: a router picks a few experts per token. You can increase total expert parameters while keeping activated expert FFN compute small, but distributed systems may pay in communication and scheduling.

LLM Systems

How models run in production: prefill vs decode, KV cache memory, batching and scheduling, and the techniques that make latency and throughput practical. Open the topic.

Decoding & Sampling: Temperature, Top-p & Inference-Time Controlfigure · code · 16 minHow inference settings reshape the next-token distribution into actual model behavior: temperature, nucleus sampling, and why decoding is a control knob.LLM Serving at Scale: Prefill, Decode & Continuous Batchingfigure · code · 22 minA systems mental model for LLM inference: prefill vs decode, TTFT vs TPOT, batching/scheduling, and why KV cache memory dominates.MoE Serving & Scheduling: Token Dispatch, All-to-All, Disaggregated Parallelismfigure · code · 20 minServing MoE turns sparse compute into a scheduling problem: routing skew can create stragglers and token-dispatch communication can bottleneck, motivating scheduling and, in systems such as MegaScale-Infer, disaggregated expert-parallel serving.Speculative Decoding: Lossless Multi-Token Generationfigure · code · 18 minDraft several tokens with a fast model, score draft prefixes with the target model in parallel, then use modified rejection/residual sampling so the sampled distribution matches target-model decoding.Structured Decoding: Token Masks From Schema Automatafigure · code · 22 minHow a schema automaton or parser state turns next-token logits into constraint-valid generation by masking invalid continuations, while leaving truth and task success outside the formal guarantee.

Standalone notes and references

AdamWConcept noteAdam (Adaptive Moment Estimation) combines the best of momentum and adaptive learning rates. It maintains both a running average of gradients (first moment) and squared gradients (second moment), adapting the learning rate for each parameter individually.Muon OptimizerConcept noteMuon is a modern optimizer designed specifically for 2D weight matrices in hidden layers (e.g. linear/attention layers in transformers). Other parameters such as embeddings and biases are typically still optimized with AdamW.Optimizers OverviewConcept noteAt the heart of every neural network lies an optimization problem. We have a loss function we want to minimize, and we need an algorithm to find the parameters that achieve this minimum. These algorithms are called optimizers .KV cache referenceReferenceCanonical reference for the KV-cache memory equation behind the open investigation: terms, regimes, and worked numbers.

Foundations

The mathematical objects underneath the models. Also readable as a connected map.

Maximum Likelihood, Cross-Entropy & KL DivergenceCore Training Scaled Dot-Product Attention & Transformer LayersCore Training Adam & Adaptive Gradient MethodsOptimization Loss Landscapes, Sharpness & Flat MinimaOptimization Overparameterization & Generalization, Double DescentOptimization Neural Tangent Kernel & Infinite-Width LimitsTheory Variational Autoencoders & Variational InferenceGenerative Models GANs & Adversarial Divergence MinimizationGenerative Models Diffusion, Score-Based Models & Flow MatchingGenerative Models Representation Learning & Embedding GeometryRepresentations Superposition, Sparse Features & MonosemanticityRepresentations Probing, Linear Classifier Probes & Activation AnalysisRepresentations Transformer Circuits, Induction Heads & Mechanistic InterpretabilityRepresentations Scaling Laws & Emergent AbilitiesScaling & Alignment Preference-Based Alignment: RLHF, Reward Modeling, Constitutional AIScaling & Alignment Efficiency: Quantization, Distillation, LoRA & Sparse MoEEfficiency Theoretical Foundations: PAC Learning, MDL & Information BottleneckTheory Efficient Attention at Scale: KV Cache, GQA & FlashAttentionEfficiency Rotary Position Embeddings (RoPE)Representations Speculative Decoding: Lossless Multi-Token GenerationEfficiency LLM Serving at Scale: Prefill, Decode & Continuous BatchingEfficiency Sparse Mixture of Experts: Routing, Load Balancing & Expert ParallelismEfficiency MoE Serving & Scheduling: Token Dispatch, All-to-All, Disaggregated ParallelismEfficiency Direct Preference Optimization: RL-Free Alignment from Human PreferencesScaling & Alignment KTO: Alignment from Binary Feedback via Human-Aware LossesScaling & Alignment Reward Hacking & Overoptimization: Goodhart's Law in Preference OptimizationScaling & Alignment Sparse Autoencoders at Scale: Feature Dictionaries for Mechanistic InterpretabilityRepresentations Automated Circuit Discovery: Patching, Attribution & Decomposition at ScaleRepresentations Activation Steering: Feature-Guided Interventions for Inference-Time ControlRepresentations Long Context Engineering: RoPE Scaling, KV Compression & Memory OptimizationEfficiency State Space Models & Hybrid Architectures: Mamba-2, Jamba, GriffinCore Training Multimodal Foundations: Vision Encoders, Contrastive Learning & Cross-Attention FusionRepresentations Tokenization & Vocabulary DesignRepresentations Decoding & Sampling: Temperature, Top-p & Inference-Time ControlCore Training Backpropagation & Automatic DifferentiationOptimization Score Matching & Score-Based Generative ModelsGenerative Models In-Context Learning: Learning Without Weight UpdatesRepresentations Optimal Transport & Wasserstein DistanceTheory Normalizing Flows: Exact Likelihood via Invertible TransformsGenerative Models PPO: Proximal Policy OptimizationScaling & Alignment Residual Connections & Skip ConnectionsCore Training Classifier-Free Guidance in DiffusionGenerative Models Retrieval-Augmented Generation (RAG)Representations Adversarial Examples & RobustnessTheory Grokking: Delayed GeneralizationTheory Logit Lens: Probing Intermediate RepresentationsRepresentations Learning Rate Schedules: Warmup, Decay & CyclingOptimization Weight Initialization: Xavier, He & µPOptimization Contrastive Learning & InfoNCERepresentations Distributed Training: Data, Tensor & Pipeline ParallelismEfficiency Beam Search & Structured DecodingCore Training Dropout: Stochastic RegularizationOptimization Energy-Based Models & Score FunctionsGenerative Models Layer Normalization & RMSNormCore Training Fisher Information & Information GeometryTheory Natural Gradient & Riemannian OptimizationOptimization SGD & Momentum: The Workhorses of OptimizationOptimization Weight Decay & AdamW: Decoupled RegularizationOptimization Gradient Clipping & Explosion PreventionOptimization Label Smoothing & Soft TargetsOptimization Batch NormalizationCore Training Knowledge Distillation: Learning from TeachersEfficiency Quantization: Compressing Models to IntegersEfficiency Pruning: Removing Unnecessary WeightsEfficiency Self-Supervised Learning: Labels from StructureRepresentations Calibration & Temperature ScalingTheory SwiGLU & Gated ActivationsCore Training FlashAttention: IO-Aware AttentionEfficiency Constitutional AI: Principles-Based AlignmentScaling & Alignment Bregman Divergence & Mirror DescentTheory Reproducing Kernel Hilbert SpacesTheory Persistent Homology & Topological Data AnalysisTheory Lie Groups & Equivariant NetworksTheory Test-Time Compute & Inference ScalingScaling & Alignment Chain-of-Thought PromptingScaling & Alignment World Models & Model-Based RLTheory Synthetic Data & Self-ImprovementScaling & Alignment Consistency Models: One-Step DiffusionGenerative Models Activation Checkpointing & Memory EfficiencyEfficiency Grouped Query Attention (GQA)Efficiency Process Reward ModelsScaling & Alignment RLAIF: AI FeedbackScaling & Alignment Flow Matching & Rectified FlowsGenerative Models Instruction TuningScaling & Alignment Deliberative AlignmentScaling & Alignment AI Safety via DebateScaling & Alignment Iterated AmplificationScaling & Alignment Weak-to-Strong GeneralizationScaling & Alignment Automated Red TeamingScaling & Alignment Mesa-Optimization & Inner AlignmentScaling & Alignment Sleeper Agents & Alignment FakingScaling & Alignment Model-Graded EvaluationsScaling & Alignment Capability Elicitation & ELKScaling & Alignment Sandwiching EvaluationsScaling & Alignment Mixture-of-DepthsEfficiency Tree Search over ThoughtsScaling & Alignment Video World ModelsGenerative Models Self-Improvement & Distillation LoopsScaling & Alignment Model Collapse & Synthetic DataTheory Infinite Context ArchitecturesEfficiency

Areas

Five long-running storylines that thread the topics together.

Sequence ModelingAreaHow models remember and retrieve context.OptimizationAreaHow training navigates the loss landscape.Generative PhysicsAreaDiffusion, flow, and the geometry of data.Geometric Deep LearningAreaSymmetry as inductive bias.Mechanistic InterpretabilityAreaReverse-engineering neural computation.

Source shelf

Books, courses, papers, notebooks, codebases, and visual explanations that help orient the learning map. These are external references, not imported course material or endorsements.

Browse all 54 referenced sources