The Atlas
Everything readable here, in one register.
Concept notes grouped by mathematical territory, the foundations underneath them, and five long-running areas that connect them. Search first if you know what you want; otherwise the whole corpus is below.
Search the AtlasLinear Algebra
Vectors, matrices, and linear maps: the language of representations, optimization, and modern deep learning. Open the topic.
Calculus
Rates of change and accumulation. Calculus is the language behind gradients, optimization, continuous-time dynamics, and why backprop works as efficiently as it does. Open the topic.
Backpropagationfigure · code · 15 minBackpropagation applies reverse-mode autodiff to neural networks so one scalar loss can train many parameters.Computation Graphsfigure · code · 13 minA computation graph breaks a calculation into nodes so values flow forward and sensitivities flow backward.Derivativesfigure · code · 14 minThe derivative is an instantaneous rate of change: the slope you get when a secant line becomes a tangent line.Reverse-Mode Automatic Differentiationfigure · code · 14 minReverse-mode autodiff computes gradients by sending cotangents backward through a computation graph.
Optimization
How we train models: gradients, learning rates, curvature, and the practical tricks that make deep nets converge. Open the topic.
Adam Optimizerfigure · code · 16 minAdam is an adaptive optimizer that combines momentum (EMA of gradients) with per-parameter RMS normalization (EMA of squared gradients).Gradient Descentfigure · code · 12 minGradient descent turns local slope information into an iterative update rule for reducing a loss.Learning Rate Schedules: Warmup, Decay & Cyclingfigure · code · 14 minSchedule shapes that change the scalar learning-rate scale over training, with sourced CLR/range-test and SGDR cosine-restart examples plus caveated warmup/decay teaching patterns.Loss Landscapes, Sharpness & Flat Minimafigure · code · 16 minHow 2D loss slices, Hessian curvature, SAM-style neighborhood loss, and a toy 2/eta stability line expose local sensitivity during optimization.
Probability
Uncertainty made precise: events, random variables, expectations, and the distributions that models learn. Open the topic.
Bayesian Inferencefigure · code · 17 minBayesian inference updates a prior distribution over unknowns into a posterior by multiplying by the likelihood and normalizing.Cross-Entropyfigure · code · 16 minCross-entropy is the target-weighted surprise of a model distribution; in deep learning it is the bridge from likelihood to a differentiable training loss.Distributionsfigure · code · 15 minA distribution is the law of a random variable: it says how probability mass or density lands on the values the variable can take.Maximum Likelihoodfigure · code · 18 minMaximum likelihood fits parameters by making the observed data most probable; for classifiers it becomes negative log-likelihood, cross-entropy, and a KL fit to the empirical distribution.Probability Basicsfigure · code · 12 minEvents are subsets of a sample space, and probabilities obey a few axioms; from there you get conditional probability, independence, and Bayes' rule.Random Variablesfigure · code · 14 minA random variable is a function from outcomes to numbers; its distribution lets you compute expectations, variances, and likelihoods.
Information Theory
How we measure information and mismatch between distributions: entropy, cross-entropy, KL divergence, mutual information, and why they appear everywhere in ML. Open the topic.
Attention & Transformers
The sequence model backbone: tokenization, self-attention, positional encodings, and the transformer block that powers modern LLMs. Open the topic.
Efficient Attention at Scale: KV Cache, GQA & FlashAttentionfigure · code · 26 minHow attention becomes practical at long context: KV caching for decoding, grouped-query attention, and IO-aware kernels like FlashAttention.FlashAttention: IO-Aware Attentionfigure · code · 18 minA fused, tiled attention implementation that avoids materializing the full T x T matrix by using an online softmax, reducing memory traffic and speeding up long-context training/inference.Layer Normalization & RMSNormfigure · code · 14 minNormalize one token/example vector across features: LayerNorm centers and scales, while RMSNorm keeps RMS-based scaling without mean-centering.Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimizationfigure · code · 22 minHow frontier LLMs stretch context windows: positional extrapolation (RoPE scaling) plus KV cache memory tricks (GQA, paging, quantization, compression).Rotary Position Embeddings (RoPE)figure · code · 14 minA positional encoding that rotates queries and keys so attention depends on relative position via phase differences.Scaled Dot-Product Attention & Transformer Layersfigure · code · 18 minThe core transformer operation: compute attention weights from query-key dot products, then mix values to copy information across a sequence.Tokenization & Vocabulary Designfigure · code · 14 minHow text becomes token IDs: segmentation, BPE/unigram tokenizers, and the tradeoffs that shape cost and capability.
Representation Learning
Embeddings and the geometry of meaning: similarity, normalization, contrastive objectives, and why vector spaces become usable interfaces for models. Open the topic.
Representation Learning & Embedding Geometryfigure · code · 18 minHow models turn inputs into vectors whose geometry can expose useful factors, contextual meaning, and similarity structure.Sparse Autoencoders: Feature Dictionaries for Mechanistic Interpretabilityfigure · code · 16 minSparse autoencoders learn a reusable dictionary of feature directions so dense model activations can be explained by a small set of interpretable latent factors.
Generative Models
How models generate: likelihood, latent variables, diffusion/score models, flows, and the training tricks that make sampling work. Open the topic.
Diffusion, Score-Based Models & Flow Matchingfigure · code · 22 minDenoise noise into data: the diffusion forward process, score matching, and modern sampling via reverse-time dynamics and flow matching.Flow Matching & Rectified Flowsfigure · code · 16 minLearn flow matching as velocity-label regression: straight conditional paths use x_t=(1-t)x0+t x1 but supervise the full target u_t=x1-x0.Normalizing Flows: Tractable Density via Invertible Transformsfigure · code · 18 minInvertible transforms trained with change-of-variables: tractable transformed densities and sampling, with architectural constraints to make the needed Jacobian determinants tractable.Score Matching & Score-Based Generative Modelsfigure · code · 18 minLearn the score field grad_x log p(x) without normalizing constants. Denoising score matching turns diffusion training into simple regression on noise.Variational Autoencodersfigure · code · 20 minA latent-variable model trained by maximizing an evidence lower bound; the gap is KL(q_phi(z|x) || p_theta(z|x)), so the encoder is learned inference rather than just compression.
Scaling
How loss and capability change with parameters, data, and compute; how to allocate a training budget; and why some abilities appear suddenly at scale. Open the topic.
Neural Tangent Kernel (NTK) & Infinite-Width Limitsfigure · code · 18 minA limit where wide neural networks behave like kernel methods: the model linearizes around initialization and training becomes kernel regression.Overparameterization & Generalization (Double Descent)figure · code · 16 minTest error can peak at the interpolation threshold then fall again as models get larger: why modern overparameterized nets still generalize.Scaling Laws & Emergent Abilitiesfigure · code · 18 minEmpirical power laws that predict how loss and capability improve with parameters, data, and compute, and how to choose compute-optimal training runs.Test-Time Compute: Spending Inference Budget on Searchfigure · code · 22 minTest-time compute spends extra inference budget on sampling, verification, and selection; it helps when the generator can produce good candidates and the verifier ranks them reliably.Tree Search Reasoning: Allocating Inference Budget Across Prefixesfigure · code · 24 minTree search spends inference budget on partial reasoning prefixes, using local verifier scores, frontier expansion, and max backups to decide which branches deserve more thought.
Alignment
How we shape model behavior: preference learning, reward modeling, KL-regularized fine-tuning, and the failure modes that appear when you optimize the wrong thing. Open the topic.
Direct Preference Optimizationfigure · code · 18 minDPO turns pairwise preferences into binary cross-entropy on reference-relative log odds, using the KL-regularized RLHF optimum to make the policy itself an implicit reward model.Kahneman-Tversky Optimizationfigure · code · 18 minKTO turns binary desirable/undesirable labels into a reference-relative utility loss: push a labeled output's policy/reference log-ratio above or below a KL-derived baseline, with saturating gradients.Process Reward Models: Step-Level Verifiers for Reasoningfigure · code · 24 minA process reward model scores intermediate reasoning steps instead of only terminal answers, giving denser verifier feedback for reranking and search while remaining a learned proxy.Reward Hacking: Overoptimizing Preference Proxiesfigure · code · 22 minWhen an imperfect preference proxy is optimized past its validation regime, policy mass shifts toward reward-model errors; KL, ensembles, LCBs, and monitoring slow this down but do not make the proxy true.RLHF: Reward Modeling + KL-Regularized Policy Optimizationfigure · code · 20 minRLHF trains a reward model from pairwise preferences, then reweights a reference policy toward high learned reward while a KL penalty limits distribution shift.
Efficiency
How we make models cheaper to train and serve: quantization, distillation, low-rank adapters, sparsity, and the memory/latency tradeoffs that dominate real deployments. Open the topic.
Efficiency: Quantization, Distillation, LoRA & Sparse MoEfigure · code · 20 minThe practical toolkit for making big models cheaper: quantize weights/activations, distill teachers into students, adapt with low-rank updates (LoRA), and use sparsity (MoE).Knowledge Distillation: Learning from Teachersfigure · code · 16 minTrain a smaller student to mimic a stronger teacher by matching soft probability distributions (often with temperature), transferring 'dark knowledge' beyond hard labels.Pruning: Removing Unnecessary Weightsfigure · code · 16 minReduce parameter count by zeroing or removing weights. Unstructured sparsity needs sparse kernels for speed; structured pruning removes whole channels/heads to shrink dense tensor shapes.Quantization: Compressing Models to Integersfigure · code · 16 minReduce memory and bandwidth by storing weights/activations in low-bit integers (INT8/INT4) with careful scaling to limit accuracy loss.Sparse Mixture of Experts: Routing, Load Balancing & Expert Parallelismfigure · code · 20 minConditional computation: a router picks a few experts per token. You can increase total expert parameters while keeping activated expert FFN compute small, but distributed systems may pay in communication and scheduling.
LLM Systems
How models run in production: prefill vs decode, KV cache memory, batching and scheduling, and the techniques that make latency and throughput practical. Open the topic.
Decoding & Sampling: Temperature, Top-p & Inference-Time Controlfigure · code · 16 minHow inference settings reshape the next-token distribution into actual model behavior: temperature, nucleus sampling, and why decoding is a control knob.LLM Serving at Scale: Prefill, Decode & Continuous Batchingfigure · code · 22 minA systems mental model for LLM inference: prefill vs decode, TTFT vs TPOT, batching/scheduling, and why KV cache memory dominates.MoE Serving & Scheduling: Token Dispatch, All-to-All, Disaggregated Parallelismfigure · code · 20 minServing MoE turns sparse compute into a scheduling problem: routing skew can create stragglers and token-dispatch communication can bottleneck, motivating scheduling and, in systems such as MegaScale-Infer, disaggregated expert-parallel serving.Speculative Decoding: Lossless Multi-Token Generationfigure · code · 18 minDraft several tokens with a fast model, score draft prefixes with the target model in parallel, then use modified rejection/residual sampling so the sampled distribution matches target-model decoding.Structured Decoding: Token Masks From Schema Automatafigure · code · 22 minHow a schema automaton or parser state turns next-token logits into constraint-valid generation by masking invalid continuations, while leaving truth and task success outside the formal guarantee.
Foundations
The mathematical objects underneath the models. Also readable as a connected map.
Maximum Likelihood, Cross-Entropy & KL DivergenceCore TrainingScaled Dot-Product Attention & Transformer LayersCore TrainingAdam & Adaptive Gradient MethodsOptimizationLoss Landscapes, Sharpness & Flat MinimaOptimizationOverparameterization & Generalization, Double DescentOptimizationNeural Tangent Kernel & Infinite-Width LimitsTheoryVariational Autoencoders & Variational InferenceGenerative ModelsGANs & Adversarial Divergence MinimizationGenerative ModelsDiffusion, Score-Based Models & Flow MatchingGenerative ModelsRepresentation Learning & Embedding GeometryRepresentationsSuperposition, Sparse Features & MonosemanticityRepresentationsProbing, Linear Classifier Probes & Activation AnalysisRepresentationsTransformer Circuits, Induction Heads & Mechanistic InterpretabilityRepresentationsScaling Laws & Emergent AbilitiesScaling & AlignmentPreference-Based Alignment: RLHF, Reward Modeling, Constitutional AIScaling & AlignmentEfficiency: Quantization, Distillation, LoRA & Sparse MoEEfficiencyTheoretical Foundations: PAC Learning, MDL & Information BottleneckTheoryEfficient Attention at Scale: KV Cache, GQA & FlashAttentionEfficiencyRotary Position Embeddings (RoPE)RepresentationsSpeculative Decoding: Lossless Multi-Token GenerationEfficiencyLLM Serving at Scale: Prefill, Decode & Continuous BatchingEfficiencySparse Mixture of Experts: Routing, Load Balancing & Expert ParallelismEfficiencyMoE Serving & Scheduling: Token Dispatch, All-to-All, Disaggregated ParallelismEfficiencyDirect Preference Optimization: RL-Free Alignment from Human PreferencesScaling & AlignmentKTO: Alignment from Binary Feedback via Human-Aware LossesScaling & AlignmentReward Hacking & Overoptimization: Goodhart's Law in Preference OptimizationScaling & AlignmentSparse Autoencoders at Scale: Feature Dictionaries for Mechanistic InterpretabilityRepresentationsAutomated Circuit Discovery: Patching, Attribution & Decomposition at ScaleRepresentationsActivation Steering: Feature-Guided Interventions for Inference-Time ControlRepresentationsLong Context Engineering: RoPE Scaling, KV Compression & Memory OptimizationEfficiencyState Space Models & Hybrid Architectures: Mamba-2, Jamba, GriffinCore TrainingMultimodal Foundations: Vision Encoders, Contrastive Learning & Cross-Attention FusionRepresentationsTokenization & Vocabulary DesignRepresentationsDecoding & Sampling: Temperature, Top-p & Inference-Time ControlCore TrainingBackpropagation & Automatic DifferentiationOptimizationScore Matching & Score-Based Generative ModelsGenerative ModelsIn-Context Learning: Learning Without Weight UpdatesRepresentationsOptimal Transport & Wasserstein DistanceTheoryNormalizing Flows: Exact Likelihood via Invertible TransformsGenerative ModelsPPO: Proximal Policy OptimizationScaling & AlignmentResidual Connections & Skip ConnectionsCore TrainingClassifier-Free Guidance in DiffusionGenerative ModelsRetrieval-Augmented Generation (RAG)RepresentationsAdversarial Examples & RobustnessTheoryGrokking: Delayed GeneralizationTheoryLogit Lens: Probing Intermediate RepresentationsRepresentationsLearning Rate Schedules: Warmup, Decay & CyclingOptimizationWeight Initialization: Xavier, He & µPOptimizationContrastive Learning & InfoNCERepresentationsDistributed Training: Data, Tensor & Pipeline ParallelismEfficiencyBeam Search & Structured DecodingCore TrainingDropout: Stochastic RegularizationOptimizationEnergy-Based Models & Score FunctionsGenerative ModelsLayer Normalization & RMSNormCore TrainingFisher Information & Information GeometryTheoryNatural Gradient & Riemannian OptimizationOptimizationSGD & Momentum: The Workhorses of OptimizationOptimizationWeight Decay & AdamW: Decoupled RegularizationOptimizationGradient Clipping & Explosion PreventionOptimizationLabel Smoothing & Soft TargetsOptimizationBatch NormalizationCore TrainingKnowledge Distillation: Learning from TeachersEfficiencyQuantization: Compressing Models to IntegersEfficiencyPruning: Removing Unnecessary WeightsEfficiencySelf-Supervised Learning: Labels from StructureRepresentationsCalibration & Temperature ScalingTheorySwiGLU & Gated ActivationsCore TrainingFlashAttention: IO-Aware AttentionEfficiencyConstitutional AI: Principles-Based AlignmentScaling & AlignmentBregman Divergence & Mirror DescentTheoryReproducing Kernel Hilbert SpacesTheoryPersistent Homology & Topological Data AnalysisTheoryLie Groups & Equivariant NetworksTheoryTest-Time Compute & Inference ScalingScaling & AlignmentChain-of-Thought PromptingScaling & AlignmentWorld Models & Model-Based RLTheorySynthetic Data & Self-ImprovementScaling & AlignmentConsistency Models: One-Step DiffusionGenerative ModelsActivation Checkpointing & Memory EfficiencyEfficiencyGrouped Query Attention (GQA)EfficiencyProcess Reward ModelsScaling & AlignmentRLAIF: AI FeedbackScaling & AlignmentFlow Matching & Rectified FlowsGenerative ModelsInstruction TuningScaling & AlignmentDeliberative AlignmentScaling & AlignmentAI Safety via DebateScaling & AlignmentIterated AmplificationScaling & AlignmentWeak-to-Strong GeneralizationScaling & AlignmentAutomated Red TeamingScaling & AlignmentMesa-Optimization & Inner AlignmentScaling & AlignmentSleeper Agents & Alignment FakingScaling & AlignmentModel-Graded EvaluationsScaling & AlignmentCapability Elicitation & ELKScaling & AlignmentSandwiching EvaluationsScaling & AlignmentMixture-of-DepthsEfficiencyTree Search over ThoughtsScaling & AlignmentVideo World ModelsGenerative ModelsSelf-Improvement & Distillation LoopsScaling & AlignmentModel Collapse & Synthetic DataTheoryInfinite Context ArchitecturesEfficiency
Areas
Five long-running storylines that thread the topics together.
Sequence ModelingAreaHow models remember and retrieve context.OptimizationAreaHow training navigates the loss landscape.Generative PhysicsAreaDiffusion, flow, and the geometry of data.Geometric Deep LearningAreaSymmetry as inductive bias.Mechanistic InterpretabilityAreaReverse-engineering neural computation.
Source shelf
Books, courses, papers, notebooks, codebases, and visual explanations that help orient the learning map. These are external references, not imported course material or endorsements.
Browse all 54 referenced sources
Training Language Models to Follow Instructions with Human FeedbackAcademic spine · canonical paper · Alignment and feedbackSupervised fine-tuning, reward modeling, PPO-style RLHF, and human preference data. Use boundary: Link to arXiv/OpenAI material; keep claims scoped to the paper and later methods. Connected vocabulary: RLHF, Reward Hacking, Process Reward Models, Direct Preference Optimization.Training Compute-Optimal Large Language ModelsAcademic spine · canonical paper · Compute-optimal trainingTradeoff between parameters, tokens, compute budget, and downstream performance. Use boundary: Link to arXiv/NeurIPS; explain as one point in a living scaling-law thread, not final scripture. Connected vocabulary: Scaling Laws, Pretraining Data Mixtures, LLM Serving, efficiency.Score-Based Generative Modeling through Stochastic Differential EquationsAcademic spine · canonical paper · Continuous-time generative modelingForward SDE, reverse-time SDE, score field, and numerical sampling. Use boundary: Link to OpenReview/arXiv; keep stochastic-process assumptions explicit. Connected vocabulary: Score Matching, Diffusion, Flow Matching, Probability Basics.Deep Residual Learning for Image RecognitionAcademic spine · canonical paper · Deep architecture optimizationResidual functions, skip connections, optimization degradation, and depth scaling. Use boundary: Use CVF open-access link; create original diagrams and explanations. Connected vocabulary: Residual Connections, Backpropagation, Representations, Attention and Transformers.Denoising Diffusion Probabilistic ModelsAcademic spine · canonical paper · Diffusion modelsForward noising process, learned reverse denoising process, variational objective, and sampling chain. Use boundary: Link to NeurIPS; use original local visuals and toy distributions. Connected vocabulary: Diffusion, Score Matching, Probability Basics, Flow Matching.FlashAttentionAcademic spine · canonical paper · Efficient attention systemsIO-aware exact attention, tiling, SRAM/HBM movement, and long-sequence scaling. Use boundary: Link to NeurIPS/OpenReview/arXiv; do not copy figures or code unless license is checked. Connected vocabulary: FlashAttention, Efficient Attention, Long Context, LLM Serving.Flow Matching for Generative ModelingAcademic spine · canonical paper · Flow-based generative modelingProbability paths, conditional vector fields, continuous normalizing flows, and simulation-free training. Use boundary: Link to OpenReview; use local low-dimensional transport demos for explanation. Connected vocabulary: Flow Matching, Normalizing Flows, Diffusion, Score Matching.Auto-Encoding Variational BayesAcademic spine · canonical paper · Latent-variable generative modelsVariational lower bound, recognition model, reparameterization trick, and amortized inference. Use boundary: Link to arXiv/ICLR record; local derivations should define every distribution and expectation. Connected vocabulary: Variational Autoencoders, Bayesian Inference, kl divergence, Maximum Likelihood.Sparse Autoencoders Find Highly Interpretable Features in Language ModelsAcademic spine · canonical paper · Mechanistic interpretabilitySparse feature dictionaries over internal activations and causal feature attribution. Use boundary: Link to arXiv/OpenReview; use local toy models before discussing real-model claims. Connected vocabulary: Sparse Autoencoders, Representations, Attention and Transformers, Scaling Laws.Adam: A Method for Stochastic OptimizationAcademic spine · canonical paper · Optimization algorithmsAdaptive first and second moment estimates for stochastic gradients. Use boundary: Link to arXiv/ICLR; keep implementation snippets minimal and notation-aligned. Connected vocabulary: Adam, Gradient Descent, learning rate schedules, Weight Decay and AdamW.RoFormer: Enhanced Transformer with Rotary Position EmbeddingAcademic spine · canonical paper · Positional structureRotations that encode absolute position while making relative position visible in attention. Use boundary: Link to arXiv and local concept pages; write original derivations and visual witnesses. Connected vocabulary: RoPE, Positional Encoding, Attention and Transformers, Long Context.Direct Preference OptimizationAcademic spine · canonical paper · Preference optimizationPreference-pair classification objective derived from reward-policy duality. Use boundary: Link to NeurIPS/OpenReview/arXiv; local math must preserve assumptions behind the derivation. Connected vocabulary: Direct Preference Optimization, RLHF, Cross-Entropy, Reward Hacking.Let's Verify Step by StepAcademic spine · canonical paper · Reasoning supervisionStep-level reward models, process supervision, active learning, and verification of reasoning traces. Use boundary: Link to OpenAI-hosted PDF; summarize source-bounded claims and avoid copying figures. Connected vocabulary: Process Reward Models, Tree Search Reasoning, RLHF, Reward Hacking.Scaling Laws for Neural Language ModelsAcademic spine · canonical paper · Scaling behaviorPower-law relations between loss, model size, dataset size, and compute. Use boundary: Link to arXiv; keep local explanations source-bounded and updated when later scaling work changes the picture. Connected vocabulary: Scaling Laws, Pretraining Data Mixtures, Cross-Entropy, Loss Landscapes.Neural Tangent KernelAcademic spine · canonical paper · Training dynamics theoryFunction-space gradient descent in the infinite-width limit. Use boundary: Link to NeurIPS/arXiv; keep the scope carefully bounded to the assumptions of the theorem. Connected vocabulary: Neural Tangent Kernel, Gradient Descent, Representations, Scaling Laws.Batch NormalizationAcademic spine · canonical paper · Training stabilizationMini-batch mean/variance normalization inside a network layer. Use boundary: Link to PMLR and cite; keep local caveats honest because later work revises the internal-covariate-shift story. Connected vocabulary: Batch Normalization, Gradient Descent, Backpropagation, Representations.Attention Is All You NeedAcademic spine · canonical paper · Transformer foundationsScaled dot-product attention, multi-head attention, positional encoding, and sequence-to-sequence architecture. Use boundary: Link to NeurIPS/arXiv pages and cite; use local diagrams and original code witnesses. Connected vocabulary: Attention and Transformers, Dot Product, Positional Encoding, Residual Connections.Gaussian Processes for Machine LearningAcademic spine · monograph · Bayesian nonparametricsDistributions over functions, kernels, marginal likelihood, posterior prediction, and uncertainty. Use boundary: The site says MIT Press allowed the web version; deep link and cite, do not redistribute chapters. Connected vocabulary: Bayesian Inference, Neural Tangent Kernel, Representations, Probability Basics.Information Theory, Inference, and Learning AlgorithmsAcademic spine · monograph · Information and inferenceEntropy, coding, Bayesian inference, graphical models, neural networks, and error-correcting codes. Use boundary: Link to the official book PDF; retain copyright boundaries and write original explanations. Connected vocabulary: Probability Basics, Bayesian Inference, Cross-Entropy, kl divergence.Convex OptimizationAcademic spine · monograph · Optimization theoryConvex sets, duality, constrained optimization, and reliable numerical procedures. Use boundary: The official site notes Cambridge allowed the book to stay available on the web; still link/cite instead of mirroring. Connected vocabulary: Gradient Descent, Loss Landscapes, Weight Decay and AdamW, Adam.Reinforcement Learning: An IntroductionAcademic spine · monograph · Reinforcement learningValue functions, policy improvement, temporal-difference learning, planning, and function approximation. Use boundary: Link to the official book page; avoid copying chapters, figures, or exercises into local pages. Connected vocabulary: RLHF, Reward Hacking, Process Reward Models, Tree Search Reasoning.In-context Learning and Induction HeadsAcademic spine · research program · In-context learning mechanismsAttention-head circuits that copy or complete repeated token patterns. Use boundary: Link to the official article/arXiv; keep local claims separated into causal, correlational, and toy evidence. Connected vocabulary: Attention and Transformers, Tokenization and Vocabulary, Representations, Sparse Autoencoders.A Mathematical Framework for Transformer CircuitsAcademic spine · research program · Mechanistic interpretabilityResidual stream, attention heads, virtual weights, paths, and circuit-level decomposition. Use boundary: Link to the Transformer Circuits thread; cite and create original diagrams/labs. Connected vocabulary: Attention and Transformers, Representations, Sparse Autoencoders, Residual Connections.Toy Models of SuperpositionAcademic spine · research program · Representation geometrySparse features represented in fewer dimensions through superposition and interference. Use boundary: Link to official thread/arXiv; build local toy demos rather than copying figures. Connected vocabulary: Sparse Autoencoders, Representations, Vector Spaces, Dot Product.Understanding Machine Learning: From Theory to AlgorithmsAcademic spine · theory text · Algorithmic learning theoryPAC learning, uniform convergence, convex learning, regularization, and online learning. Use boundary: Prefer official author/course links; cite the book and write local concept-specific bridges. Connected vocabulary: Gradient Descent, Loss Landscapes, Maximum Likelihood, Representations.Foundations of Machine LearningAcademic spine · theory text · Learning theoryGeneralization bounds, Rademacher complexity, kernels, online learning, and boosting. Use boundary: Use the official author page and MIT Press online versions; avoid reproducing text or exercises. Connected vocabulary: Scaling Laws, Loss Landscapes, Gradient Descent, Representations.Pattern Recognition and Machine LearningAcademic spine · theory text · Probabilistic MLBayesian decision theory, graphical models, kernels, mixture models, variational inference, and EM. Use boundary: Use official Springer or author-hosted links where available; do not mirror PDFs from unofficial repositories. Connected vocabulary: Bayesian Inference, Maximum Likelihood, Normalizing Flows, Variational Autoencoders.The Elements of Statistical LearningAcademic spine · theory text · Statistical learning foundationsBias-variance, regularization paths, kernels, trees, boosting, and unsupervised structure. Use boundary: Use the official author page as the link target; cite and synthesize rather than copying text or figures. Connected vocabulary: Maximum Likelihood, Cross-Entropy, Representations, Loss Landscapes.Dive into Deep LearningResearch-learning · book · Interactive bookDeep learning from math to code across PyTorch, JAX, TensorFlow, and NumPy-style implementations. Use boundary: Deep link, cite, and synthesize. Do not copy chapters, figures, or code blocks into canonical pages. Connected vocabulary: Gradient Descent, Backpropagation, Attention and Transformers, Diffusion.Mathematics for Machine LearningResearch-learning · book · Math bridgeLinear algebra, analytic geometry, matrix decompositions, vector calculus, probability, and optimization. Use boundary: Link to the official book site; respect personal-use PDF language and avoid redistribution. Connected vocabulary: Vector Spaces, Dot Product, Derivatives, Gradient Descent.Understanding Deep LearningResearch-learning · book · Modern deep-learning textA pragmatic treatment of deep learning concepts, modern architectures, and training behavior. Use boundary: Deep link to official material and cite; keep Continuous Function explanations original. Connected vocabulary: Backpropagation, Adam, Attention and Transformers, Diffusion.Probabilistic Machine LearningResearch-learning · book · Probabilistic modeling referenceModern probabilistic ML, Bayesian modeling, uncertainty, graphical models, and deep generative links. Use boundary: Link to official resources; do not mirror book text or figures unless license permits. Connected vocabulary: Probability Basics, Bayesian Inference, Maximum Likelihood, Score Matching.Deep LearningResearch-learning · book · Reference textbookMathematical and conceptual background for deep learning, optimization, probability, and architectures. Use boundary: Link to the official online book and cite specific chapters; do not redistribute derived excerpts. Connected vocabulary: Probability Basics, Maximum Likelihood, Gradient Descent, Representations.OpenAI CookbookResearch-learning · codebase · Applied API examplesPractical examples and guides for building with modern AI APIs. Use boundary: Link to examples and docs; respect repository license before reusing code. Connected vocabulary: Retrieval-Augmented Generation, Structured Decoding, Decoding and Sampling, LLM Serving.TransformerLensResearch-learning · codebase · Mechanistic interpretability toolkitPractical tools, tutorials, and exercises for inspecting transformer activations and circuits. Use boundary: Link to docs and repo; respect code license before reuse. Connected vocabulary: Attention and Transformers, Sparse Autoencoders, Representations, Scaling Laws.Berkeley CS285 Deep Reinforcement LearningResearch-learning · course · Deep RL courseModern RL methods, offline-to-online RL, LLM RL, and project-driven RL practice. Use boundary: Link to official materials; avoid copying assignments or lecture assets. Connected vocabulary: RLHF, Reward Hacking, Process Reward Models, Tree Search Reasoning.Stanford CS231n Deep Learning for Computer VisionResearch-learning · course · Deep-learning systems courseTraining neural networks, optimization, backpropagation, CNNs, debugging, and visual recognition. Use boundary: Link to official notes and materials; do not copy assignments or solutions into the site. Connected vocabulary: Backpropagation, Gradient Descent, Adam, Representations.Stanford CS336 Language Modeling from ScratchResearch-learning · course · Frontier language-model courseThe process of building language models from tokenizer and data to training, scaling, and evaluation. Use boundary: Link to current and archived official offerings; do not mirror assignments or course files. Connected vocabulary: Tokenization and Vocabulary, Attention and Transformers, Scaling Laws, Pretraining Data Mixtures.Berkeley Deep Unsupervised LearningResearch-learning · course · Generative and self-supervised learning courseDeep generative models, self-supervised learning, diffusion, flows, and modern representation learning. Use boundary: Link to official course pages; do not copy slides, assignments, or videos. Connected vocabulary: Diffusion, Score Matching, Flow Matching, Representations.OpenAI Spinning Up in Deep RLResearch-learning · course · Implementation-oriented RL guideDeep RL concepts, algorithms, exercises, failure modes, and benchmarks. Use boundary: Link to official docs and repo; cite; keep local code witnesses minimal and original. Connected vocabulary: RLHF, Reward Hacking, Process Reward Models, Tree Search Reasoning.Stanford CS229 Machine LearningResearch-learning · course · ML foundations courseClassical ML, supervised learning, generalization, optimization, kernels, and probabilistic models. Use boundary: Link and cite official course pages or notes; keep local derivations original. Connected vocabulary: Maximum Likelihood, Cross-Entropy, Gradient Descent, Bayesian Inference.Stanford CS224N NLP with Deep LearningResearch-learning · course · NLP and LLM courseWord vectors, sequence models, attention, transformers, and modern LLM topics. Use boundary: Deep link and cite official lectures/readings; do not reproduce course assets without permission. Connected vocabulary: Attention and Transformers, Positional Encoding, RoPE, Decoding and Sampling.MIT OCW 18.06 Linear AlgebraResearch-learning · course · Open coursewareMatrix theory, vector spaces, eigenvalues, positive definite matrices, and useful applications. Use boundary: Use official OCW links and license-aware attribution; avoid copying lecture assets wholesale. Connected vocabulary: Vector Spaces, Dot Product, Linear Transformations, Representations.Hugging Face LearnResearch-learning · course · Open ecosystem curriculumLLMs, transformers, agents, diffusion, deep RL, computer vision, and practical libraries. Use boundary: Link to course chapters and docs; avoid copying exercises, datasets, or model cards. Connected vocabulary: Attention and Transformers, Decoding and Sampling, Diffusion, RLHF.fast.ai Practical Deep Learning for CodersResearch-learning · course · Top-down practical coursePractical model building with fast feedback and runnable notebooks. Use boundary: Link to the course and notebooks; do not copy lesson text or notebooks into canonical content. Connected vocabulary: Representations, Backpropagation, Gradient Descent, Diffusion.Full Stack Deep LearningResearch-learning · engineering guide · AI product engineeringProblem formulation, data, training, reproducibility, deployment, and product considerations. Use boundary: Link to course pages and materials; keep local systems guidance original. Connected vocabulary: LLM Serving, Retrieval-Augmented Generation, Structured Decoding, efficiency.Made With MLResearch-learning · engineering guide · Production ML guideDesigning, developing, deploying, and iterating production ML applications. Use boundary: Link to lessons and repo; do not copy prose or code wholesale. Connected vocabulary: LLM Serving, Knowledge Distillation, Quantization, Pretraining Data Mixtures.The Annotated TransformerResearch-learning · notebook · Annotated implementationLine-by-line implementation of the Transformer paper as a working notebook. Use boundary: Link to the post/repo; use only license-compatible snippets with attribution if ever needed. Connected vocabulary: Attention and Transformers, Positional Encoding, Cross-Entropy, Gradient Descent.UvA Deep Learning TutorialsResearch-learning · notebook · Notebook coursePyTorch-oriented tutorial notebooks for optimization, transformers, graph neural networks, and more. Use boundary: Deep link to notebooks and docs; cite; keep local examples small and original. Connected vocabulary: Gradient Descent, Attention and Transformers, Representations, Sparse Autoencoders.Transformer CircuitsResearch-learning · paper thread · Mechanistic interpretability research threadReverse-engineering transformer circuits, attention heads, features, and language-model internals. Use boundary: Deep link to articles; cite; keep local interpretability demos original and bounded. Connected vocabulary: Attention and Transformers, Sparse Autoencoders, Representations, Scaling Laws.Jay Alammar visual explanationsResearch-learning · visual essay · Accessible mechanism visualsTransformers, retrieval, embeddings, and LLM mechanisms through clear diagrams. Use boundary: Link to posts; use as visual pedagogy inspiration only unless reuse rights are explicit. Connected vocabulary: Attention and Transformers, Retrieval-Augmented Generation, Tokenization and Vocabulary, Decoding and Sampling.colah's blogResearch-learning · visual essay · Conceptual visual essaysNeural-network intuition, topology, recurrent nets, convolutions, attention, and interpretability. Use boundary: Link and cite; do not copy figures or prose without explicit license support. Connected vocabulary: Representations, Attention and Transformers, Sparse Autoencoders, Vector Spaces.DistillResearch-learning · visual essay · Interactive research communicationVisual, interactive explanations of neural networks, interpretability, and optimization. Use boundary: Check article-level licenses before reuse; prefer citations and original repo-native visuals. Connected vocabulary: Representations, Sparse Autoencoders, Loss Landscapes, Attention and Transformers.Lil'LogResearch-learning · visual essay · Research synthesis blogCareful long-form syntheses across transformers, diffusion, RL, alignment, agents, and inference. Use boundary: Link and cite posts; write original summaries and maintain source boundaries. Connected vocabulary: Diffusion, RLHF, Reward Hacking, Long Context.