The Atlas
Everything readable here, in one register.
Concept notes grouped by mathematical territory, the foundations underneath them, and five long-running areas that connect them. Search first if you know what you want; otherwise the whole corpus is below.
Linear Algebra
Vectors, matrices, and linear maps: the language of representations, optimization, and modern deep learning. Open the topic.
Calculus
Rates of change and accumulation. Calculus is the language behind gradients, optimization, continuous-time dynamics, and why backprop works as efficiently as it does. Open the topic.
Backpropagationfigure · code · 15 minBackpropagation applies reverse-mode autodiff to neural networks so one scalar loss can train many parameters.Computation Graphsfigure · code · 13 minA computation graph breaks a calculation into nodes so values flow forward and sensitivities flow backward.Derivativesfigure · code · 14 minThe derivative is an instantaneous rate of change: the slope you get when a secant line becomes a tangent line.Reverse-Mode Automatic Differentiationfigure · code · 14 minReverse-mode autodiff computes gradients by sending cotangents backward through a computation graph.
Optimization
How we train models: gradients, learning rates, curvature, and the practical tricks that make deep nets converge. Open the topic.
Adam Optimizerfigure · code · 16 minAdam is an adaptive optimizer that combines momentum (EMA of gradients) with per-parameter RMS normalization (EMA of squared gradients).Gradient Descentfigure · code · 12 minGradient descent turns local slope information into an iterative update rule for reducing a loss.Learning Rate Schedules: Warmup, Decay & Cyclingfigure · code · 14 minSchedule shapes that change the scalar learning-rate scale over training, with sourced CLR/range-test and SGDR cosine-restart examples plus caveated warmup/decay teaching patterns.Loss Landscapes, Sharpness & Flat Minimafigure · code · 16 minHow 2D loss slices, Hessian curvature, SAM-style neighborhood loss, and a toy 2/eta stability line expose local sensitivity during optimization.
Probability
Uncertainty made precise: events, random variables, expectations, and the distributions that models learn. Open the topic.
Bayesian Inferencefigure · code · 17 minBayesian inference updates a prior distribution over unknowns into a posterior by multiplying by the likelihood and normalizing.Cross-Entropyfigure · code · 16 minCross-entropy is the target-weighted surprise of a model distribution; in deep learning it is the bridge from likelihood to a differentiable training loss.Distributionsfigure · code · 15 minA distribution is the law of a random variable: it says how probability mass or density lands on the values the variable can take.Maximum Likelihoodfigure · code · 18 minMaximum likelihood fits parameters by making the observed data most probable; for classifiers it becomes negative log-likelihood, cross-entropy, and a KL fit to the empirical distribution.Probability Basicsfigure · code · 12 minEvents are subsets of a sample space, and probabilities obey a few axioms; from there you get conditional probability, independence, and Bayes' rule.Random Variablesfigure · code · 14 minA random variable is a function from outcomes to numbers; its distribution lets you compute expectations, variances, and likelihoods.
Information Theory
How we measure information and mismatch between distributions: entropy, cross-entropy, KL divergence, mutual information, and why they appear everywhere in ML. Open the topic.
Attention & Transformers
The sequence model backbone: tokenization, self-attention, positional encodings, and the transformer block that powers modern LLMs. Open the topic.
Efficient Attention at Scale: KV Cache, GQA & FlashAttentionfigure · code · 26 minHow attention becomes practical at long context: KV caching for decoding, grouped-query attention, and IO-aware kernels like FlashAttention.FlashAttention: IO-Aware Attentionfigure · code · 18 minA fused, tiled attention implementation that avoids materializing the full T x T matrix by using an online softmax, reducing memory traffic and speeding up long-context training/inference.Layer Normalization & RMSNormfigure · code · 14 minNormalize one token/example vector across features: LayerNorm centers and scales, while RMSNorm keeps RMS-based scaling without mean-centering.Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimizationfigure · code · 22 minHow frontier LLMs stretch context windows: positional extrapolation (RoPE scaling) plus KV cache memory tricks (GQA, paging, quantization, compression).Rotary Position Embeddings (RoPE)figure · code · 14 minA positional encoding that rotates queries and keys so attention depends on relative position via phase differences.Scaled Dot-Product Attention & Transformer Layersfigure · code · 18 minThe core transformer operation: compute attention weights from query-key dot products, then mix values to copy information across a sequence.Tokenization & Vocabulary Designfigure · code · 14 minHow text becomes token IDs: segmentation, BPE/unigram tokenizers, and the tradeoffs that shape cost and capability.
Representation Learning
Embeddings and the geometry of meaning: similarity, normalization, contrastive objectives, and why vector spaces become usable interfaces for models. Open the topic.
Representation Learning & Embedding Geometryfigure · code · 18 minHow models turn inputs into vectors whose geometry can expose useful factors, contextual meaning, and similarity structure.Sparse Autoencoders: Feature Dictionaries for Mechanistic Interpretabilityfigure · code · 16 minSparse autoencoders learn a reusable dictionary of feature directions so dense model activations can be explained by a small set of interpretable latent factors.
Generative Models
How models generate: likelihood, latent variables, diffusion/score models, flows, and the training tricks that make sampling work. Open the topic.
Diffusion, Score-Based Models & Flow Matchingfigure · code · 22 minDenoise noise into data: the diffusion forward process, score matching, and modern sampling via reverse-time dynamics and flow matching.Flow Matching & Rectified Flowsfigure · code · 16 minLearn flow matching as velocity-label regression: straight conditional paths use x_t=(1-t)x0+t x1 but supervise the full target u_t=x1-x0.Normalizing Flows: Tractable Density via Invertible Transformsfigure · code · 18 minInvertible transforms trained with change-of-variables: tractable transformed densities and sampling, with architectural constraints to make the needed Jacobian determinants tractable.Score Matching & Score-Based Generative Modelsfigure · code · 18 minLearn the score field grad_x log p(x) without normalizing constants. Denoising score matching turns diffusion training into simple regression on noise.Variational Autoencodersfigure · code · 20 minA latent-variable model trained by maximizing an evidence lower bound; the gap is KL(q_phi(z|x) || p_theta(z|x)), so the encoder is learned inference rather than just compression.
Scaling
How loss and capability change with parameters, data, and compute; how to allocate a training budget; and why some abilities appear suddenly at scale. Open the topic.
Neural Tangent Kernel (NTK) & Infinite-Width Limitsfigure · code · 18 minA limit where wide neural networks behave like kernel methods: the model linearizes around initialization and training becomes kernel regression.Overparameterization & Generalization (Double Descent)figure · code · 16 minTest error can peak at the interpolation threshold then fall again as models get larger: why modern overparameterized nets still generalize.Scaling Laws & Emergent Abilitiesfigure · code · 18 minEmpirical power laws that predict how loss and capability improve with parameters, data, and compute, and how to choose compute-optimal training runs.Test-Time Compute: Spending Inference Budget on Searchfigure · code · 22 minTest-time compute spends extra inference budget on sampling, verification, and selection; it helps when the generator can produce good candidates and the verifier ranks them reliably.Tree Search Reasoning: Allocating Inference Budget Across Prefixesfigure · code · 24 minTree search spends inference budget on partial reasoning prefixes, using local verifier scores, frontier expansion, and max backups to decide which branches deserve more thought.
Alignment
How we shape model behavior: preference learning, reward modeling, KL-regularized fine-tuning, and the failure modes that appear when you optimize the wrong thing. Open the topic.
Direct Preference Optimizationfigure · code · 18 minDPO turns pairwise preferences into binary cross-entropy on reference-relative log odds, using the KL-regularized RLHF optimum to make the policy itself an implicit reward model.Kahneman-Tversky Optimizationfigure · code · 18 minKTO turns binary desirable/undesirable labels into a reference-relative utility loss: push a labeled output's policy/reference log-ratio above or below a KL-derived baseline, with saturating gradients.Process Reward Models: Step-Level Verifiers for Reasoningfigure · code · 24 minA process reward model scores intermediate reasoning steps instead of only terminal answers, giving denser verifier feedback for reranking and search while remaining a learned proxy.Reward Hacking: Overoptimizing Preference Proxiesfigure · code · 22 minWhen an imperfect preference proxy is optimized past its validation regime, policy mass shifts toward reward-model errors; KL, ensembles, LCBs, and monitoring slow this down but do not make the proxy true.RLHF: Reward Modeling + KL-Regularized Policy Optimizationfigure · code · 20 minRLHF trains a reward model from pairwise preferences, then reweights a reference policy toward high learned reward while a KL penalty limits distribution shift.
Efficiency
How we make models cheaper to train and serve: quantization, distillation, low-rank adapters, sparsity, and the memory/latency tradeoffs that dominate real deployments. Open the topic.
Efficiency: Quantization, Distillation, LoRA & Sparse MoEfigure · code · 20 minThe practical toolkit for making big models cheaper: quantize weights/activations, distill teachers into students, adapt with low-rank updates (LoRA), and use sparsity (MoE).Knowledge Distillation: Learning from Teachersfigure · code · 16 minTrain a smaller student to mimic a stronger teacher by matching soft probability distributions (often with temperature), transferring 'dark knowledge' beyond hard labels.Pruning: Removing Unnecessary Weightsfigure · code · 16 minReduce parameter count by zeroing or removing weights. Unstructured sparsity needs sparse kernels for speed; structured pruning removes whole channels/heads to shrink dense tensor shapes.Quantization: Compressing Models to Integersfigure · code · 16 minReduce memory and bandwidth by storing weights/activations in low-bit integers (INT8/INT4) with careful scaling to limit accuracy loss.Sparse Mixture of Experts: Routing, Load Balancing & Expert Parallelismfigure · code · 20 minConditional computation: a router picks a few experts per token. You can increase total expert parameters while keeping activated expert FFN compute small, but distributed systems may pay in communication and scheduling.
LLM Systems
How models run in production: prefill vs decode, KV cache memory, batching and scheduling, and the techniques that make latency and throughput practical. Open the topic.
Decoding & Sampling: Temperature, Top-p & Inference-Time Controlfigure · code · 16 minHow inference settings reshape the next-token distribution into actual model behavior: temperature, nucleus sampling, and why decoding is a control knob.LLM Serving at Scale: Prefill, Decode & Continuous Batchingfigure · code · 22 minA systems mental model for LLM inference: prefill vs decode, TTFT vs TPOT, batching/scheduling, and why KV cache memory dominates.MoE Serving & Scheduling: Token Dispatch, All-to-All, Disaggregated Parallelismfigure · code · 20 minServing MoE turns sparse compute into a scheduling problem: routing skew can create stragglers and token-dispatch communication can bottleneck, motivating scheduling and, in systems such as MegaScale-Infer, disaggregated expert-parallel serving.Speculative Decoding: Lossless Multi-Token Generationfigure · code · 18 minDraft several tokens with a fast model, score draft prefixes with the target model in parallel, then use modified rejection/residual sampling so the sampled distribution matches target-model decoding.Structured Decoding: Token Masks From Schema Automatafigure · code · 22 minHow a schema automaton or parser state turns next-token logits into constraint-valid generation by masking invalid continuations, while leaving truth and task success outside the formal guarantee.
Foundations
The mathematical objects underneath the models. Also readable as a connected map.
Maximum Likelihood, Cross-Entropy & KL DivergenceCore TrainingScaled Dot-Product Attention & Transformer LayersCore TrainingAdam & Adaptive Gradient MethodsOptimizationLoss Landscapes, Sharpness & Flat MinimaOptimizationOverparameterization & Generalization, Double DescentOptimizationNeural Tangent Kernel & Infinite-Width LimitsTheoryVariational Autoencoders & Variational InferenceGenerative ModelsGANs & Adversarial Divergence MinimizationGenerative ModelsDiffusion, Score-Based Models & Flow MatchingGenerative ModelsRepresentation Learning & Embedding GeometryRepresentationsSuperposition, Sparse Features & MonosemanticityRepresentationsProbing, Linear Classifier Probes & Activation AnalysisRepresentationsTransformer Circuits, Induction Heads & Mechanistic InterpretabilityRepresentationsScaling Laws & Emergent AbilitiesScaling & AlignmentPreference-Based Alignment: RLHF, Reward Modeling, Constitutional AIScaling & AlignmentEfficiency: Quantization, Distillation, LoRA & Sparse MoEEfficiencyTheoretical Foundations: PAC Learning, MDL & Information BottleneckTheoryEfficient Attention at Scale: KV Cache, GQA & FlashAttentionEfficiencyRotary Position Embeddings (RoPE)RepresentationsSpeculative Decoding: Lossless Multi-Token GenerationEfficiencyLLM Serving at Scale: Prefill, Decode & Continuous BatchingEfficiencySparse Mixture of Experts: Routing, Load Balancing & Expert ParallelismEfficiencyMoE Serving & Scheduling: Token Dispatch, All-to-All, Disaggregated ParallelismEfficiencyDirect Preference Optimization: RL-Free Alignment from Human PreferencesScaling & AlignmentKTO: Alignment from Binary Feedback via Human-Aware LossesScaling & AlignmentReward Hacking & Overoptimization: Goodhart's Law in Preference OptimizationScaling & AlignmentSparse Autoencoders at Scale: Feature Dictionaries for Mechanistic InterpretabilityRepresentationsAutomated Circuit Discovery: Patching, Attribution & Decomposition at ScaleRepresentationsActivation Steering: Feature-Guided Interventions for Inference-Time ControlRepresentationsLong Context Engineering: RoPE Scaling, KV Compression & Memory OptimizationEfficiencyState Space Models & Hybrid Architectures: Mamba-2, Jamba, GriffinCore TrainingMultimodal Foundations: Vision Encoders, Contrastive Learning & Cross-Attention FusionRepresentationsTokenization & Vocabulary DesignRepresentationsDecoding & Sampling: Temperature, Top-p & Inference-Time ControlCore TrainingBackpropagation & Automatic DifferentiationOptimizationScore Matching & Score-Based Generative ModelsGenerative ModelsIn-Context Learning: Learning Without Weight UpdatesRepresentationsOptimal Transport & Wasserstein DistanceTheoryNormalizing Flows: Exact Likelihood via Invertible TransformsGenerative ModelsPPO: Proximal Policy OptimizationScaling & AlignmentResidual Connections & Skip ConnectionsCore TrainingClassifier-Free Guidance in DiffusionGenerative ModelsRetrieval-Augmented Generation (RAG)RepresentationsAdversarial Examples & RobustnessTheoryGrokking: Delayed GeneralizationTheoryLogit Lens: Probing Intermediate RepresentationsRepresentationsLearning Rate Schedules: Warmup, Decay & CyclingOptimizationWeight Initialization: Xavier, He & µPOptimizationContrastive Learning & InfoNCERepresentationsDistributed Training: Data, Tensor & Pipeline ParallelismEfficiencyBeam Search & Structured DecodingCore TrainingDropout: Stochastic RegularizationOptimizationEnergy-Based Models & Score FunctionsGenerative ModelsLayer Normalization & RMSNormCore TrainingFisher Information & Information GeometryTheoryNatural Gradient & Riemannian OptimizationOptimizationSGD & Momentum: The Workhorses of OptimizationOptimizationWeight Decay & AdamW: Decoupled RegularizationOptimizationGradient Clipping & Explosion PreventionOptimizationLabel Smoothing & Soft TargetsOptimizationBatch NormalizationCore TrainingKnowledge Distillation: Learning from TeachersEfficiencyQuantization: Compressing Models to IntegersEfficiencyPruning: Removing Unnecessary WeightsEfficiencySelf-Supervised Learning: Labels from StructureRepresentationsCalibration & Temperature ScalingTheorySwiGLU & Gated ActivationsCore TrainingFlashAttention: IO-Aware AttentionEfficiencyConstitutional AI: Principles-Based AlignmentScaling & AlignmentBregman Divergence & Mirror DescentTheoryReproducing Kernel Hilbert SpacesTheoryPersistent Homology & Topological Data AnalysisTheoryLie Groups & Equivariant NetworksTheoryTest-Time Compute & Inference ScalingScaling & AlignmentChain-of-Thought PromptingScaling & AlignmentWorld Models & Model-Based RLTheorySynthetic Data & Self-ImprovementScaling & AlignmentConsistency Models: One-Step DiffusionGenerative ModelsActivation Checkpointing & Memory EfficiencyEfficiencyGrouped Query Attention (GQA)EfficiencyProcess Reward ModelsScaling & AlignmentRLAIF: AI FeedbackScaling & AlignmentFlow Matching & Rectified FlowsGenerative ModelsInstruction TuningScaling & AlignmentDeliberative AlignmentScaling & AlignmentAI Safety via DebateScaling & AlignmentIterated AmplificationScaling & AlignmentWeak-to-Strong GeneralizationScaling & AlignmentAutomated Red TeamingScaling & AlignmentMesa-Optimization & Inner AlignmentScaling & AlignmentSleeper Agents & Alignment FakingScaling & AlignmentModel-Graded EvaluationsScaling & AlignmentCapability Elicitation & ELKScaling & AlignmentSandwiching EvaluationsScaling & AlignmentMixture-of-DepthsEfficiencyTree Search over ThoughtsScaling & AlignmentVideo World ModelsGenerative ModelsSelf-Improvement & Distillation LoopsScaling & AlignmentModel Collapse & Synthetic DataTheoryInfinite Context ArchitecturesEfficiency
Areas
Five long-running storylines that thread the topics together.
Sequence ModelingAreaHow models remember and retrieve context.OptimizationAreaHow training navigates the loss landscape.Generative PhysicsAreaDiffusion, flow, and the geometry of data.Geometric Deep LearningAreaSymmetry as inductive bias.Mechanistic InterpretabilityAreaReverse-engineering neural computation.