Index of concepts
This index lists the notebooks, foundation pages, standalone notes and sources on the site, and arranges the notebooks in 12 topics, from linear algebra to LLM systems. Search finds a concept by name. The curriculum presents the same notebooks in teaching order.
Search the notebooks, foundations, standalone notes and sources by name.
New to attention? Start here
How do comparisons become a mixture?
Follow the calculation: vectors → compare → normalize → mix
Follow one query through attention. Aim to explain how scores become weights, and how those weights carry information from values into an output.
One query attending to three tokens, with no mask. The worked example is open; no prediction is required first.
Vectors
Query, keys, values
Give the query and each key matching coordinates. Keep the values as the information to carry.
Read Vector SpacesQuery + keys → comparisons
Compare
Scaled scores
Multiply matching query–key coordinates, add, then divide by the square root of the key dimension.
Read Dot ProductScores → normalization
Normalize
Attention weights
Softmax turns the score row into nonnegative weights whose sum is one.
Inspect Weights in the notebookWeights + values → mixture
Mix
Output vector
Multiply each value by its weight, then add those contributions. The output lives in value space.
Inspect Mixture in the notebookOne weighted output
Normalize and Mix open the same notebook. Select Weights or Mixture there; these are views of one calculation, not separate lessons. Query and key vectors share a dimension; values may have a different dimension.
Need a foundation? Review just the missing idea
Optional reading, not a required course or a readiness test. Each concept page lists its own prerequisites.
- What can I add and scale?Vector Spaces
- How do two vectors become a score?Dot Product
- What does a total weight of one mean?Probability Basics
The notebook also offers Practice and transfer and Advanced demonstrations; choose them inside the notebook. Opening a route here does not record progress or establish mastery.
By topic
56 notebooks · 12 topics · 100 foundations · 4 standalone notes · 54 sources
Linear Algebra
Vectors, matrices, and linear maps: the language of representations, optimization, and modern deep learning. Open the topic.
Dot Productfigure · code · 12 minInteractive figureThe dot product measures alignment: it connects angles, lengths, and projections, and underlies cosine similarity in ML.Vector Spacesfigure · code · 10 minInteractive figureA vector space is a set of objects you can add and scale, where those operations behave consistently.
Calculus
Rates of change and how they combine: derivatives, the chain rule, Jacobians and Taylor approximation, and the reverse-mode differentiation that makes backpropagation efficient. Open the topic.
Backpropagationfigure · code · 15 minInteractive figureBackpropagation applies reverse-mode autodiff to neural networks so one scalar loss can train many parameters.Computation Graphsfigure · code · 13 minInteractive figureA computation graph breaks a calculation into nodes so values flow forward and sensitivities flow backward.Derivativesfigure · code · 14 minInteractive figureThe derivative is an instantaneous rate of change: the slope you get when a secant line becomes a tangent line.Reverse-Mode Automatic Differentiationfigure · code · 14 minInteractive figureReverse-mode autodiff computes gradients by sending sensitivities (cotangents) backward through a computation graph, so one backward pass gives the gradient of a scalar loss.
Optimization
How neural networks are trained: gradient steps, learning rates and schedules, curvature and sharpness, and adaptive optimizers such as Adam. Open the topic.
Adam Optimizerfigure · code · 16 minInteractive figureAdam combines momentum, a running average of gradients, with a per-parameter step size scaled by a running average of squared gradients.Gradient Clipping & Explosion Preventionfigure · code · 13 minInteractive figureNorm clipping rescales gradients above a threshold while preserving their direction. It bounds a plain gradient-descent step, but the same bound need not hold for an adaptive optimizer that rescales coordinates or averages earlier gradients.Gradient Descentfigure · code · 20 minInteractive figureGradient descent turns local slope information into an iterative update rule for reducing a loss.Learning Rate Schedules: Warmup, Decay & Cyclingfigure · code · 14 minInteractive figureHow warmup, decay and cyclical schedules change the learning rate over a training run, including the Transformer's warmup with inverse-square-root decay and cosine annealing.Loss Landscapes, Sharpness & Flat Minimafigure · code · 16 minInteractive figureWhat 2D slices of a loss surface show and hide, how Hessian curvature measures sharpness, why the link between flat minima and generalization is contested, and how SAM trains toward flat regions.Weight Decay & AdamW: Decoupled Regularizationfigure · code · 13 minInteractive figureWhy shrinking weights is not the same as adding an L2 penalty inside Adam, and how AdamW restores the intended regularization behavior.
Probability
Events, random variables, expectations and the distributions used in machine learning. Open the topic.
Bayesian Inferencefigure · code · 17 minInteractive figureBayesian inference updates a prior distribution over unknowns into a posterior by multiplying by the likelihood and normalizing.Cross-Entropyfigure · code · 16 minInteractive figureCross-entropy is the target-weighted surprise of a model distribution; in deep learning it is the bridge from likelihood to a differentiable training loss.Distributionsfigure · code · 15 minInteractive figureA distribution is the law of a random variable: it says how probability mass or density lands on the values the variable can take.Maximum Likelihoodfigure · code · 18 minInteractive figureMaximum likelihood fits parameters by making the observed data most probable; for classifiers it becomes negative log-likelihood, cross-entropy, and a KL fit to the empirical distribution.Probability Basicsfigure · code · 12 minInteractive figureEvents are subsets of a sample space, and probabilities obey a few axioms; from there you get conditional probability, independence, and Bayes' rule.Random Variablesfigure · code · 14 minInteractive figureA random variable is a function from outcomes to numbers; its distribution lets you compute expectations, variances, and likelihoods.
Information Theory
How information and the mismatch between distributions are measured: entropy, cross-entropy and KL divergence, and why they appear throughout machine learning. Open the topic.
Attention & Transformers
The transformer architecture: tokenization, self-attention, positional encodings and transformer blocks, with calculations of attention and its memory cost at long context. Open the topic.
Efficient Attention at Scale: KV Cache, GQA & FlashAttentionfigure · code · 26 minInteractive figureHow attention becomes practical at long context: KV caching for decoding, grouped-query attention, and IO-aware kernels like FlashAttention.FlashAttention: IO-Aware Attentionfigure · code · 30 minInteractive figureExact attention computed tile by tile with a running softmax, so the full T × T matrix is never written to GPU main memory; in the paper's benchmarks the reduced memory traffic made attention faster.Layer Normalization & RMSNormfigure · code · 24 minInteractive figureNormalizing each token's vector across its features: LayerNorm subtracts the mean and divides by the standard deviation; RMSNorm only divides by the root mean square.Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimizationfigure · code · 22 minInteractive figureHow models handle long inputs: keeping positions within the range seen in training (RoPE scaling), sliding windows, and cutting KV-cache memory with GQA, paging, quantization and compression.Positional Encodingfigure · code · 12 minInteractive figureHow transformers represent order: sinusoidal encodings, learned embeddings, and relative-position methods like RoPE.Residual Connections & Skip Connectionsfigure · code · 14 minInteractive figureA residual connection adds a layer's output to its input, giving the input a direct path through the network while the layer learns an update.Rotary Position Embeddings (RoPE)figure · code · 14 minInteractive figureA positional encoding that rotates queries and keys so attention depends on relative position via phase differences.Scaled Dot-Product Attention & Transformer Layersfigure · code · 18 minInteractive figureHow a query scores keys with scaled dot products, turns the scores into softmax weights and mixes the values; with masks, several queries at once and the transformer layer around them.SwiGLU: Gated MLP Blocks in Transformersfigure · code · 16 minInteractive figureWhy modern transformer MLPs often use a learned multiplicative gate: one projection proposes a token-local write while another projection controls how much of each channel reaches the residual stream.Tokenization & Vocabulary Designfigure · code · 14 minInteractive figureHow text becomes token IDs: segmentation, BPE/unigram tokenizers, and the tradeoffs that shape cost and capability.
Representation Learning
How models represent inputs as vectors: similarity and normalization, contrastive training, and sparse autoencoders for reading features out of activations. Open the topic.
Representation Learning & Embedding Geometryfigure · code · 18 minInteractive figureHow models turn inputs into vectors whose geometry can expose useful factors, contextual meaning, and similarity structure.Sparse Autoencoders: Feature Dictionaries for Mechanistic Interpretabilityfigure · code · 16 minInteractive figureSparse autoencoders rewrite each model activation as a sparse combination of learned directions, which are often easier to interpret than individual neurons.
Generative Models
How generative models are built and trained: latent-variable models and the ELBO, normalizing flows, diffusion and score-based models, and flow matching. Open the topic.
Diffusion, Score-Based Models & Flow Matchingfigure · code · 25 minInteractive figureHow diffusion models generate: a fixed forward process adds Gaussian noise, a network learns to predict that noise, and a sampler runs the process in reverse.Flow Matching & Rectified Flowsfigure · code · 34 minInteractive figureFlow matching trains a velocity field by regression: on the straight path x_t = (1-t)x0 + t x1 the label is x1 - x0, and the learned field is the average label E[x1 - x0 | x_t].Normalizing Flows: Tractable Density via Invertible Transformsfigure · code · 22 minInteractive figureInvertible maps that turn a simple density into a richer one: exact log-likelihood from the change-of-variables formula, and layer designs that keep the Jacobian determinant cheap.Score Matching & Score-Based Generative Modelsfigure · code · 21 minInteractive figureLearn the score, the gradient of log p(x), without the normalizing constant. Denoising score matching turns this into regression on the added noise, which is how diffusion models are trained.Variational Autoencodersfigure · code · 20 minInteractive figureA latent-variable model trained by maximizing the evidence lower bound (ELBO). The bound falls short of the log-likelihood by the KL divergence from the encoder to the true posterior, so the encoder learns approximate inference, not just compression.
Scaling
How loss and capability change with parameters, data and compute; how to allocate a training budget; why some abilities seem to appear suddenly at scale; and how to spend compute at inference time. Open the topic.
Neural Tangent Kernel (NTK) & Infinite-Width Limitsfigure · code · 18 minInteractive figureIn the infinite-width limit under the NTK parameterization, squared-loss training by gradient flow or with a sufficiently small learning rate follows the network's linearization at initialization. When the training kernel is invertible, the converged predictions add a kernel-regression fit of the initial errors to the initial predictions. The notebook contrasts this fixed-kernel regime with feature learning.Overparameterization & Generalization (Double Descent)figure · code · 16 minInteractive figureTest error can peak at the interpolation threshold, where a model first fits its training data exactly, and fall again as the model grows past it.Scaling Laws & Emergent Abilitiesfigure · code · 18 minInteractive figureEmpirical power laws for how pretraining loss falls with parameters, data and compute, how to split a fixed budget between model size and tokens, and why some abilities seem to appear suddenly.Test-Time Compute: Spending Inference Budget on Searchfigure · code · 60 minInteractive figureTest-time compute spends extra inference budget on sampling, verification, and selection; it helps when the generator can produce good candidates and the verifier ranks them reliably.Tree Search Reasoning: Allocating Inference Budget Across Prefixesfigure · code · 32 minInteractive figureTree search spends inference budget on partial reasoning prefixes, using local verifier scores, frontier expansion, and max backups to decide which branches deserve more thought.
Alignment
Training language models from human feedback: reward models fit to comparisons, KL-regularized fine-tuning, direct preference methods, step-level verifiers, and what goes wrong when a learned proxy is optimized too hard. Open the topic.
Direct Preference Optimizationfigure · code · 18 minInteractive figureDPO turns pairwise preferences into binary cross-entropy on reference-relative log odds, using the KL-regularized RLHF optimum to make the policy itself an implicit reward model.Kahneman-Tversky Optimizationfigure · code · 18 minInteractive figureKTO turns binary desirable/undesirable labels into a reference-relative utility loss: push a labeled output's policy/reference log-ratio above or below a KL-derived baseline, with saturating gradients.Process Reward Models: Step-Level Verifiers for Reasoningfigure · code · 15 minInteractive figureA process reward model scores intermediate reasoning steps rather than only the final answer. These learned scores can guide reranking and search, but remain proxies for correctness.Reward Hacking: Overoptimizing Preference Proxiesfigure · code · 22 minInteractive figureWhen an imperfect learned reward is optimized beyond the range where it was validated, probability shifts toward the reward model's errors; KL penalties, ensembles, lower-confidence-bound scores and monitoring slow this down but do not make the proxy true.RLHF: Reward Modeling + KL-Regularized Policy Optimizationfigure · code · 20 minInteractive figureRLHF trains a reward model from pairwise preferences, then reweights a reference policy toward high learned reward while a KL penalty limits distribution shift.
Efficiency
Ways to make models cheaper to train and serve: quantization, distillation, low-rank adapters, pruning and sparse experts, and the memory, latency and accuracy each one trades. Open the topic.
Efficiency: Quantization, Distillation, LoRA & Sparse MoEfigure · code · 20 minInteractive figureFour common ways to make large models cheaper to run or adapt: quantize weights and activations, distill a teacher into a smaller student, train a low-rank update (LoRA) or route each token to a few experts (sparse MoE).Knowledge Distillation: Learning from Teachersfigure · code · 16 minInteractive figureKnowledge distillation trains a student, usually smaller, to match a teacher's temperature-softened output probabilities, often alongside the true labels. The probabilities the teacher gives to wrong classes show which classes it treats as similar, which hard labels omit.Pruning Weights and Channelsfigure · code · 20 minInteractive figureReduce parameter count by zeroing or removing weights. Unstructured sparsity needs sparse kernels for speed; structured pruning removes whole channels/heads to shrink dense tensor shapes.Quantization: Compressing Models to Integersfigure · code · 16 minInteractive figureReduce memory and bandwidth by storing weights/activations in low-bit integers (INT8/INT4) with careful scaling to limit accuracy loss.Sparse Mixture of Experts: Routing, Load Balancing & Expert Parallelismfigure · code · 20 minInteractive figureConditional computation: a router picks a few experts per token. You can increase total expert parameters while keeping activated expert FFN compute small, but distributed systems may pay in communication and scheduling.
LLM Systems
How language models are served: prefill and decode, KV-cache memory, batching and scheduling, decoding rules, and techniques such as speculative decoding that cut latency. Open the topic.
Decoding & Sampling: Temperature, Top-p & Inference-Time Controlfigure · code · 16 minInteractive figureHow decoding settings reshape the next-token distribution before a token is drawn: temperature, nucleus (top-p) sampling, and why the decoding rule changes what a model writes.LLM Serving at Scale: Prefill, Decode & Continuous Batchingfigure · code · 22 minInteractive figureHow LLM inference is served: prefill and decode, time to first token and time per output token, batching and scheduling, and why KV-cache memory often limits batch size.MoE Serving & Scheduling: Token Dispatch, All-to-All, Disaggregated Parallelismfigure · code · 20 minInteractive figureServing a mixture-of-experts model turns sparse computation into a scheduling problem: uneven routing creates stragglers and sending tokens to experts costs communication, which motivates careful scheduling and, in systems such as MegaScale-Infer, separate GPU pools for attention and experts.Speculative Decoding: Lossless Multi-Token Generationfigure · code · 18 minInteractive figureDraft several tokens with a fast model, score all the draft prefixes with the target model in one parallel pass, then use modified rejection sampling so the output distribution matches the target model's.Structured Decoding: Token Masks From Schema Automatafigure · code · 22 minInteractive figureHow a schema automaton or parser state turns next-token logits into constraint-valid generation by masking invalid continuations, while leaving truth and task success outside the formal guarantee.
Standalone notes
AdamWConcept noteAdamW is Adam with weight decay applied directly to the weights, outside Adam's per-parameter scaling (Loshchilov & Hutter, 2019).Muon OptimizerConcept noteMuon (Jordan et al., 2024) orthogonalizes the momentum update for a hidden layer’s two-dimensional weight matrix, such as a transformer’s linear or attention projection.Optimizers OverviewConcept noteAn optimizer is a rule for moving a model's parameters from wₜ to wₜ₊₁ so as to reduce the loss L. The optimizers below (gradient descent, momentum, and adaptive updates such as RMSProp) differ mainly in two choices: the direction of each step and its size.KV cache referenceReferenceWhat a transformer’s key–value cache stores, the formula for its memory with a worked example, and how grouped-query and multi-query attention shrink it.
Foundations
100 earlier pages, written before the notebooks, on topics from the mathematics of training to alignment and interpretability. They also have their own index.
How to use these
Open one when a notebook relies on an idea you want to look up. They do not need to be read in order.
A row marked Newer notebook opens the earlier page; a newer notebook on the same topic is listed above, under its topic.
Maximum Likelihood, Cross-Entropy & KL DivergenceCore TrainingNewer notebookScaled Dot-Product Attention & Transformer LayersCore TrainingNewer notebookAdam & Adaptive Gradient MethodsOptimizationNewer notebookLoss Landscapes, Sharpness & Flat MinimaOptimizationNewer notebookOverparameterization & Generalization, Double DescentOptimizationNewer notebookNeural Tangent Kernel & Infinite-Width LimitsTheoryNewer notebookVariational Autoencoders & Variational InferenceGenerative ModelsNewer notebookGANs & Adversarial Divergence MinimizationGenerative ModelsReadingDiffusion, Score-Based Models & Flow MatchingGenerative ModelsNewer notebookRepresentation Learning & Embedding GeometryRepresentationsNewer notebookSuperposition, Sparse Features & MonosemanticityRepresentationsReadingProbing, Linear Classifier Probes & Activation AnalysisRepresentationsReadingTransformer Circuits, Induction Heads & Mechanistic InterpretabilityRepresentationsReadingScaling Laws & Emergent AbilitiesScaling & AlignmentNewer notebookPreference-Based Alignment: RLHF, Reward Modeling, Constitutional AIScaling & AlignmentNewer notebookEfficiency: Quantization, Distillation, LoRA & Sparse MoEEfficiencyNewer notebookTheoretical Foundations: PAC Learning, MDL & Information BottleneckTheoryReadingEfficient Attention at Scale: KV Cache, GQA & FlashAttentionEfficiencyNewer notebookRotary Position Embeddings (RoPE)RepresentationsNewer notebookSpeculative Decoding: Lossless Multi-Token GenerationEfficiencyNewer notebookLLM Serving at Scale: Prefill, Decode & Continuous BatchingEfficiencyNewer notebookSparse Mixture of Experts: Routing, Load Balancing & Expert ParallelismEfficiencyNewer notebookMoE Serving & Scheduling: Token Dispatch, All-to-All, Disaggregated ParallelismEfficiencyNewer notebookDirect Preference Optimization: RL-Free Alignment from Human PreferencesScaling & AlignmentNewer notebookKTO: Alignment from Binary Feedback via Human-Aware LossesScaling & AlignmentNewer notebookReward Hacking & Overoptimization: Goodhart's Law in Preference OptimizationScaling & AlignmentNewer notebookSparse Autoencoders at Scale: Feature Dictionaries for Mechanistic InterpretabilityRepresentationsNewer notebookAutomated Circuit Discovery: Patching, Attribution & Decomposition at ScaleRepresentationsReadingActivation Steering: Feature-Guided Interventions for Inference-Time ControlRepresentationsReadingLong Context Engineering: RoPE Scaling, KV Compression & Memory OptimizationEfficiencyNewer notebookState Space Models & Hybrid Architectures: Mamba-2, Jamba, GriffinCore TrainingReadingMultimodal Foundations: Vision Encoders, Contrastive Learning & Cross-Attention FusionRepresentationsReadingTokenization & Vocabulary DesignRepresentationsNewer notebookDecoding & Sampling: Temperature, Top-p & Inference-Time ControlCore TrainingNewer notebookBackpropagation & Automatic DifferentiationOptimizationReadingScore Matching & Score-Based Generative ModelsGenerative ModelsNewer notebookIn-Context Learning: Learning Without Weight UpdatesRepresentationsReadingOptimal Transport & Wasserstein DistanceTheoryReadingNormalizing Flows: Exact Likelihood via Invertible TransformsGenerative ModelsNewer notebookPPO: Proximal Policy OptimizationScaling & AlignmentReadingResidual Connections & Skip ConnectionsCore TrainingNewer notebookClassifier-Free Guidance in DiffusionGenerative ModelsReadingRetrieval-Augmented Generation (RAG)RepresentationsReadingAdversarial Examples & RobustnessTheoryReadingGrokking: Delayed GeneralizationTheoryReadingLogit Lens: Probing Intermediate RepresentationsRepresentationsReadingLearning Rate Schedules: Warmup, Decay & CyclingOptimizationNewer notebookWeight Initialization: Xavier, He & µPOptimizationReadingContrastive Learning & InfoNCERepresentationsReadingDistributed Training: Data, Tensor & Pipeline ParallelismEfficiencyReadingBeam Search & Structured DecodingCore TrainingReadingDropout: Stochastic RegularizationOptimizationReadingEnergy-Based Models & Score FunctionsGenerative ModelsReadingLayer Normalization & RMSNormCore TrainingNewer notebookFisher Information & Information GeometryTheoryReadingNatural Gradient & Riemannian OptimizationOptimizationReadingStochastic Gradient Descent & MomentumOptimizationReadingWeight Decay & AdamW: Decoupled RegularizationOptimizationNewer notebookGradient Clipping & Explosion PreventionOptimizationNewer notebookLabel Smoothing & Soft TargetsOptimizationReadingBatch NormalizationCore TrainingReadingKnowledge Distillation: Learning from TeachersEfficiencyNewer notebookQuantization: Compressing Models to IntegersEfficiencyNewer notebookPruning: Removing Unnecessary WeightsEfficiencyNewer notebookSelf-Supervised Learning: Labels from StructureRepresentationsReadingCalibration & Temperature ScalingTheoryReadingSwiGLU & Gated ActivationsCore TrainingNewer notebookFlashAttention: IO-Aware AttentionEfficiencyNewer notebookConstitutional AI: Principles-Based AlignmentScaling & AlignmentReadingBregman Divergence & Mirror DescentTheoryReadingReproducing Kernel Hilbert SpacesTheoryReadingPersistent Homology & Topological Data AnalysisTheoryReadingLie Groups & Equivariant NetworksTheoryReadingTest-Time Compute & Inference ScalingScaling & AlignmentNewer notebookChain-of-Thought PromptingScaling & AlignmentReadingWorld Models & Model-Based RLTheoryReadingSynthetic Training DataScaling & AlignmentReadingConsistency ModelsGenerative ModelsReadingActivation Checkpointing & Memory EfficiencyEfficiencyReadingGrouped-Query Attention (GQA)EfficiencyReadingProcess Reward ModelsScaling & AlignmentNewer notebookRLAIF: Reinforcement Learning from AI FeedbackScaling & AlignmentReadingFlow Matching & Rectified FlowsGenerative ModelsNewer notebookInstruction TuningScaling & AlignmentReadingDeliberative AlignmentScaling & AlignmentReadingAI Safety via DebateScaling & AlignmentReadingIterated AmplificationScaling & AlignmentReadingWeak-to-Strong GeneralizationScaling & AlignmentReadingAutomated Red TeamingScaling & AlignmentReadingMesa-Optimization & Inner AlignmentScaling & AlignmentReadingSleeper Agents & Alignment FakingScaling & AlignmentReadingModel-Graded EvaluationsScaling & AlignmentReadingCapability Elicitation & Eliciting Latent KnowledgeScaling & AlignmentReadingSandwiching EvaluationsScaling & AlignmentReadingMixture-of-DepthsEfficiencyReadingTree Search for Language-Model ReasoningScaling & AlignmentNewer notebookVideo World ModelsGenerative ModelsReadingSelf-Improvement & Distillation LoopsScaling & AlignmentReadingModel CollapseTheoryReadingInfinite-Context ArchitecturesEfficiencyReading
Areas
Five overview pages, each following one theme across several topics.
Sequence ModelingAreaHow models remember and retrieve context.OptimizationAreaHow training moves across the loss landscape.Generative PhysicsAreaDiffusion, flow, and the geometry of data.Geometric Deep LearningAreaSymmetry as inductive bias.Mechanistic InterpretabilityAreaReverse-engineering neural computation.
Source shelf
Books, courses, papers, notebooks, code and visual explanations that the curriculum draws on or points to. They are external references, not material copied here and not endorsements.
Reuse
Each row says how the site uses the source. A link does not grant permission to copy, redistribute, train on, or adapt the source; reusing any part of it means checking its license first.
Browse all 54 referenced sources
Training Language Models to Follow Instructions with Human FeedbackBook or paper · canonical paper · Alignment and feedbackSupervised fine-tuning, reward modeling, PPO-style RLHF, and human preference data. How the site uses it: Link to arXiv/OpenAI material; keep claims scoped to the paper and later methods. Related topics: RLHF, Reward Hacking, Process Reward Models, Direct Preference Optimization.Training Compute-Optimal Large Language ModelsBook or paper · canonical paper · Compute-optimal trainingTradeoff between parameters, tokens, compute budget, and downstream performance. How the site uses it: Link to arXiv/NeurIPS; present it as one result in continuing work on scaling laws, not the final word. Related topics: Scaling Laws, Pretraining Data Mixtures, LLM Serving, efficiency.Score-Based Generative Modeling through Stochastic Differential EquationsBook or paper · canonical paper · Continuous-time generative modelingForward SDE, reverse-time SDE, score field, and numerical sampling. How the site uses it: Link to OpenReview/arXiv; keep stochastic-process assumptions explicit. Related topics: Score Matching, Diffusion, Flow Matching, Probability Basics.Deep Residual Learning for Image RecognitionBook or paper · canonical paper · Deep architecture optimizationResidual functions, skip connections, optimization degradation, and depth scaling. How the site uses it: Use CVF open-access link; create original diagrams and explanations. Related topics: Residual Connections, Backpropagation, Representations, Attention and Transformers.Denoising Diffusion Probabilistic ModelsBook or paper · canonical paper · Diffusion modelsForward noising process, learned reverse denoising process, variational objective, and sampling chain. How the site uses it: Link to NeurIPS; use original local visuals and toy distributions. Related topics: Diffusion, Score Matching, Probability Basics, Flow Matching.FlashAttentionBook or paper · canonical paper · Efficient attention systemsIO-aware exact attention, tiling, SRAM/HBM movement, and long-sequence scaling. How the site uses it: Link to NeurIPS/OpenReview/arXiv; do not copy figures or code unless license is checked. Related topics: FlashAttention, Efficient Attention, Long Context, LLM Serving.Flow Matching for Generative ModelingBook or paper · canonical paper · Flow-based generative modelingProbability paths, conditional vector fields, continuous normalizing flows, and simulation-free training. How the site uses it: Link to OpenReview; use local low-dimensional transport demos for explanation. Related topics: Flow Matching, Normalizing Flows, Diffusion, Score Matching.Auto-Encoding Variational BayesBook or paper · canonical paper · Latent-variable generative modelsVariational lower bound, recognition model, reparameterization trick, and amortized inference. How the site uses it: Link to arXiv/ICLR record; local derivations should define every distribution and expectation. Related topics: Variational Autoencoders, Bayesian Inference, kl divergence, Maximum Likelihood.Sparse Autoencoders Find Highly Interpretable Features in Language ModelsBook or paper · canonical paper · Mechanistic interpretabilitySparse feature dictionaries over internal activations and causal feature attribution. How the site uses it: Link to arXiv/OpenReview; use local toy models before discussing real-model claims. Related topics: Sparse Autoencoders, Representations, Attention and Transformers, Scaling Laws.Adam: A Method for Stochastic OptimizationBook or paper · canonical paper · Optimization algorithmsAdaptive first and second moment estimates for stochastic gradients. How the site uses it: Link to arXiv/ICLR; keep implementation snippets minimal and notation-aligned. Related topics: Adam, Gradient Descent, learning rate schedules, Weight Decay and AdamW.RoFormer: Enhanced Transformer with Rotary Position EmbeddingBook or paper · canonical paper · Positional structureRotations that encode absolute position while making relative position visible in attention. How the site uses it: Link to arXiv and to the related notebooks on this site; derivations and figures here are our own. Related topics: RoPE, Positional Encoding, Attention and Transformers, Long Context.Direct Preference OptimizationBook or paper · canonical paper · Preference optimizationPreference-pair classification objective derived from reward-policy duality. How the site uses it: Link to NeurIPS/OpenReview/arXiv; local math must preserve assumptions behind the derivation. Related topics: Direct Preference Optimization, RLHF, Cross-Entropy, Reward Hacking.Let's Verify Step by StepBook or paper · canonical paper · Reasoning supervisionStep-level reward models, process supervision, active learning, and verification of reasoning traces. How the site uses it: Link to the PDF hosted by OpenAI; summarize only what the paper claims, and do not copy figures. Related topics: Process Reward Models, Tree Search Reasoning, RLHF, Reward Hacking.Scaling Laws for Neural Language ModelsBook or paper · canonical paper · Scaling behaviorPower-law relations between loss, model size, dataset size, and compute. How the site uses it: Link to arXiv; explanations here keep to what the paper shows and are updated when later scaling work changes the picture. Related topics: Scaling Laws, Pretraining Data Mixtures, Cross-Entropy, Loss Landscapes.Neural Tangent KernelBook or paper · canonical paper · Training dynamics theoryFunction-space gradient descent in the infinite-width limit. How the site uses it: Link to NeurIPS/arXiv; state the result only under the assumptions of the theorem. Related topics: Neural Tangent Kernel, Gradient Descent, Representations, Scaling Laws.Batch NormalizationBook or paper · canonical paper · Training stabilizationMini-batch mean/variance normalization inside a network layer. How the site uses it: Link to PMLR and cite; keep local caveats honest because later work revises the internal-covariate-shift story. Related topics: Batch Normalization, Gradient Descent, Backpropagation, Representations.Attention Is All You NeedBook or paper · canonical paper · Transformer foundationsScaled dot-product attention, multi-head attention, positional encoding, and sequence-to-sequence architecture. How the site uses it: Link to the NeurIPS and arXiv pages and cite the paper; diagrams and code on this site are our own. Related topics: Attention and Transformers, Dot Product, Positional Encoding, Residual Connections.Gaussian Processes for Machine LearningBook or paper · monograph · Bayesian nonparametricsDistributions over functions, kernels, marginal likelihood, posterior prediction, and uncertainty. How the site uses it: The site says MIT Press allowed the web version; deep link and cite, do not redistribute chapters. Related topics: Bayesian Inference, Neural Tangent Kernel, Representations, Probability Basics.Information Theory, Inference, and Learning AlgorithmsBook or paper · monograph · Information and inferenceEntropy, coding, Bayesian inference, graphical models, neural networks, and error-correcting codes. How the site uses it: Link to the official book PDF; respect its copyright and write explanations of our own. Related topics: Probability Basics, Bayesian Inference, Cross-Entropy, kl divergence.Convex OptimizationBook or paper · monograph · Optimization theoryConvex sets, duality, constrained optimization, and reliable numerical procedures. How the site uses it: The official site notes Cambridge allowed the book to stay available on the web; still link/cite instead of mirroring. Related topics: Gradient Descent, Loss Landscapes, Weight Decay and AdamW, Adam.Reinforcement Learning: An IntroductionBook or paper · monograph · Reinforcement learningValue functions, policy improvement, temporal-difference learning, planning, and function approximation. How the site uses it: Link to the official book page; avoid copying chapters, figures, or exercises into local pages. Related topics: RLHF, Reward Hacking, Process Reward Models, Tree Search Reasoning.In-context Learning and Induction HeadsBook or paper · research program · In-context learning mechanismsAttention-head circuits that copy or complete repeated token patterns. How the site uses it: Link to the official article/arXiv; keep local claims separated into causal, correlational, and toy evidence. Related topics: Attention and Transformers, Tokenization and Vocabulary, Representations, Sparse Autoencoders.A Mathematical Framework for Transformer CircuitsBook or paper · research program · Mechanistic interpretabilityResidual stream, attention heads, virtual weights, paths, and circuit-level decomposition. How the site uses it: Link to the Transformer Circuits thread; cite and create original diagrams/labs. Related topics: Attention and Transformers, Representations, Sparse Autoencoders, Residual Connections.Toy Models of SuperpositionBook or paper · research program · Representation geometrySparse features represented in fewer dimensions through superposition and interference. How the site uses it: Link to official thread/arXiv; build local toy demos rather than copying figures. Related topics: Sparse Autoencoders, Representations, Vector Spaces, Dot Product.Understanding Machine Learning: From Theory to AlgorithmsBook or paper · theory text · Algorithmic learning theoryPAC learning, uniform convergence, convex learning, regularization, and online learning. How the site uses it: Prefer official author or course links; cite the book and write our own explanations for each notebook that uses it. Related topics: Gradient Descent, Loss Landscapes, Maximum Likelihood, Representations.Foundations of Machine LearningBook or paper · theory text · Learning theoryGeneralization bounds, Rademacher complexity, kernels, online learning, and boosting. How the site uses it: Use the official author page and MIT Press online versions; avoid reproducing text or exercises. Related topics: Scaling Laws, Loss Landscapes, Gradient Descent, Representations.Pattern Recognition and Machine LearningBook or paper · theory text · Probabilistic MLBayesian decision theory, graphical models, kernels, mixture models, variational inference, and EM. How the site uses it: Use official Springer or author-hosted links where available; do not mirror PDFs from unofficial repositories. Related topics: Bayesian Inference, Maximum Likelihood, Normalizing Flows, Variational Autoencoders.The Elements of Statistical LearningBook or paper · theory text · Statistical learning foundationsBias-variance, regularization paths, kernels, trees, boosting, and unsupervised structure. How the site uses it: Use the official author page as the link target; cite and synthesize rather than copying text or figures. Related topics: Maximum Likelihood, Cross-Entropy, Representations, Loss Landscapes.Dive into Deep LearningLearning resource · book · Interactive bookDeep learning from math to code across PyTorch, JAX, TensorFlow, and NumPy-style implementations. How the site uses it: Deep link, cite, and synthesize. Do not copy chapters, figures, or code blocks into pages on this site. Related topics: Gradient Descent, Backpropagation, Attention and Transformers, Diffusion.Mathematics for Machine LearningLearning resource · book · Math bridgeLinear algebra, analytic geometry, matrix decompositions, vector calculus, probability, and optimization. How the site uses it: Link to the official book site; respect personal-use PDF language and avoid redistribution. Related topics: Vector Spaces, Dot Product, Derivatives, Gradient Descent.Understanding Deep LearningLearning resource · book · Modern deep-learning textA pragmatic treatment of deep learning concepts, modern architectures, and training behavior. How the site uses it: Deep link to official material and cite; keep Continuous Function explanations original. Related topics: Backpropagation, Adam, Attention and Transformers, Diffusion.Probabilistic Machine LearningLearning resource · book · Probabilistic modeling referenceModern probabilistic ML, Bayesian modeling, uncertainty, graphical models, and deep generative links. How the site uses it: Link to official resources; do not mirror book text or figures unless license permits. Related topics: Probability Basics, Bayesian Inference, Maximum Likelihood, Score Matching.Deep LearningLearning resource · book · Reference textbookMathematical and conceptual background for deep learning, optimization, probability, and architectures. How the site uses it: Link to the official online book and cite specific chapters; do not redistribute derived excerpts. Related topics: Probability Basics, Maximum Likelihood, Gradient Descent, Representations.OpenAI CookbookLearning resource · codebase · Applied API examplesPractical examples and guides for building with modern AI APIs. How the site uses it: Link to examples and docs; respect repository license before reusing code. Related topics: Retrieval-Augmented Generation, Structured Decoding, Decoding and Sampling, LLM Serving.TransformerLensLearning resource · codebase · Mechanistic interpretability toolkitPractical tools, tutorials, and exercises for inspecting transformer activations and circuits. How the site uses it: Link to docs and repo; respect code license before reuse. Related topics: Attention and Transformers, Sparse Autoencoders, Representations, Scaling Laws.Berkeley CS285 Deep Reinforcement LearningLearning resource · course · Deep RL courseModern RL methods, offline-to-online RL, LLM RL, and project-driven RL practice. How the site uses it: Link to official materials; avoid copying assignments or lecture assets. Related topics: RLHF, Reward Hacking, Process Reward Models, Tree Search Reasoning.Stanford CS231n Deep Learning for Computer VisionLearning resource · course · Deep-learning systems courseTraining neural networks, optimization, backpropagation, CNNs, debugging, and visual recognition. How the site uses it: Link to official notes and materials; do not copy assignments or solutions into the site. Related topics: Backpropagation, Gradient Descent, Adam, Representations.Stanford CS336 Language Modeling from ScratchLearning resource · course · Frontier language-model courseThe process of building language models from tokenizer and data to training, scaling, and evaluation. How the site uses it: Link to current and archived official offerings; do not mirror assignments or course files. Related topics: Tokenization and Vocabulary, Attention and Transformers, Scaling Laws, Pretraining Data Mixtures.Berkeley Deep Unsupervised LearningLearning resource · course · Generative and self-supervised learning courseDeep generative models, self-supervised learning, diffusion, flows, and modern representation learning. How the site uses it: Link to official course pages; do not copy slides, assignments, or videos. Related topics: Diffusion, Score Matching, Flow Matching, Representations.OpenAI Spinning Up in Deep RLLearning resource · course · Implementation-oriented RL guideDeep RL concepts, algorithms, exercises, failure modes, and benchmarks. How the site uses it: Link to the official docs and repository and cite them; any code on this site is short and our own. Related topics: RLHF, Reward Hacking, Process Reward Models, Tree Search Reasoning.Stanford CS229 Machine LearningLearning resource · course · ML foundations courseClassical ML, supervised learning, generalization, optimization, kernels, and probabilistic models. How the site uses it: Link and cite official course pages or notes; keep local derivations original. Related topics: Maximum Likelihood, Cross-Entropy, Gradient Descent, Bayesian Inference.Stanford CS224N NLP with Deep LearningLearning resource · course · NLP and LLM courseWord vectors, sequence models, attention, transformers, and modern LLM topics. How the site uses it: Deep link and cite official lectures/readings; do not reproduce course assets without permission. Related topics: Attention and Transformers, Positional Encoding, RoPE, Decoding and Sampling.MIT OCW 18.06 Linear AlgebraLearning resource · course · Open coursewareMatrix theory, vector spaces, eigenvalues, positive definite matrices, and useful applications. How the site uses it: Use official OCW links and license-aware attribution; avoid copying lecture assets wholesale. Related topics: Vector Spaces, Dot Product, Linear Transformations, Representations.Hugging Face LearnLearning resource · course · Open ecosystem curriculumLLMs, transformers, agents, diffusion, deep RL, computer vision, and practical libraries. How the site uses it: Link to course chapters and docs; avoid copying exercises, datasets, or model cards. Related topics: Attention and Transformers, Decoding and Sampling, Diffusion, RLHF.fast.ai Practical Deep Learning for CodersLearning resource · course · Top-down practical coursePractical model building with fast feedback and runnable notebooks. How the site uses it: Link to the course and notebooks; do not copy lesson text or notebooks onto this site. Related topics: Representations, Backpropagation, Gradient Descent, Diffusion.Full Stack Deep LearningLearning resource · engineering guide · AI product engineeringProblem formulation, data, training, reproducibility, deployment, and product considerations. How the site uses it: Link to course pages and materials; keep local systems guidance original. Related topics: LLM Serving, Retrieval-Augmented Generation, Structured Decoding, efficiency.Made With MLLearning resource · engineering guide · Production ML guideDesigning, developing, deploying, and iterating production ML applications. How the site uses it: Link to lessons and repo; do not copy prose or code wholesale. Related topics: LLM Serving, Knowledge Distillation, Quantization, Pretraining Data Mixtures.The Annotated TransformerLearning resource · notebook · Annotated implementationLine-by-line implementation of the Transformer paper as a working notebook. How the site uses it: Link to the post/repo; use only license-compatible snippets with attribution if ever needed. Related topics: Attention and Transformers, Positional Encoding, Cross-Entropy, Gradient Descent.UvA Deep Learning TutorialsLearning resource · notebook · Notebook coursePyTorch-oriented tutorial notebooks for optimization, transformers, graph neural networks, and more. How the site uses it: Deep link to notebooks and docs; cite; keep local examples small and original. Related topics: Gradient Descent, Attention and Transformers, Representations, Sparse Autoencoders.Transformer CircuitsLearning resource · paper thread · Mechanistic interpretability research threadReverse-engineering transformer circuits, attention heads, features, and language-model internals. How the site uses it: Link to and cite the articles; interpretability demos on this site are our own small examples. Related topics: Attention and Transformers, Sparse Autoencoders, Representations, Scaling Laws.Jay Alammar visual explanationsLearning resource · visual essay · Accessible mechanism visualsTransformers, retrieval, embeddings, and LLM mechanisms through clear diagrams. How the site uses it: Link to posts; use as visual pedagogy inspiration only unless reuse rights are explicit. Related topics: Attention and Transformers, Retrieval-Augmented Generation, Tokenization and Vocabulary, Decoding and Sampling.colah's blogLearning resource · visual essay · Conceptual visual essaysNeural-network intuition, topology, recurrent nets, convolutions, attention, and interpretability. How the site uses it: Link and cite; do not copy figures or prose without explicit license support. Related topics: Representations, Attention and Transformers, Sparse Autoencoders, Vector Spaces.DistillLearning resource · visual essay · Interactive research communicationVisual, interactive explanations of neural networks, interpretability, and optimization. How the site uses it: Check each article’s license before reusing anything; prefer citing it and drawing our own figures. Related topics: Representations, Sparse Autoencoders, Loss Landscapes, Attention and Transformers.Lil'LogLearning resource · visual essay · Research synthesis blogCareful long-form syntheses across transformers, diffusion, RL, alignment, agents, and inference. How the site uses it: Link to and cite the posts; summaries on this site are our own and keep clear which claims come from the post. Related topics: Diffusion, RLHF, Reward Hacking, Long Context.