Domain
Scaling
How loss and capability change with parameters, data and compute; how to allocate a training budget; why some abilities seem to appear suddenly at scale; and how to spend compute at inference time.
Preview the recommended route
Start here. Make one prediction, then note what carries over to the next notebook.
NDClosspower lawLearnerWhich notebook should I read first?
Open the first notebookRead the first notebook, make one prediction in its demo, then go on to the next.
Recommended route
Start here, then follow the prerequisites forward.
Read these in order: each notebook comes after the notebooks in this domain that it builds on, and the label beside it gives its level. A prerequisite with no page here yet is marked as outside reading. A notebook is left out of the route only when a prerequisite has neither a page here nor a named outside reading; the full list below marks it.
- 01Overparameterization & Generalization (Double Descent)
Test error can peak at the interpolation threshold, where a model first fits its training data exactly, and fall again as the model grows past it.
Intermediate16 mincodedemoafter Loss Landscapes, Sharpness & Flat Minima - 02Scaling Laws & Emergent Abilities
Empirical power laws for how pretraining loss falls with parameters, data and compute, how to split a fixed budget between model size and tokens, and why some abilities seem to appear suddenly.
Intermediate18 mincodedemoafter Scaled Dot-Product Attention & Transformer Layers, Overparameterization & Generalization (Double Descent)Scaling Laws & Emergent Abilities builds directly on Overparameterization & Generalization.
- 03Test-Time Compute: Spending Inference Budget on Search
Test-time compute spends extra inference budget on sampling, verification, and selection; it helps when the generator can produce good candidates and the verifier ranks them reliably.
Advanced60 mincodedemoafter Probability Basics - 04Neural Tangent Kernel (NTK) & Infinite-Width Limits
In the infinite-width limit under the NTK parameterization, squared-loss training by gradient flow or with a sufficiently small learning rate follows the network's linearization at initialization. When the training kernel is invertible, the converged predictions add a kernel-regression fit of the initial errors to the initial predictions. The notebook contrasts this fixed-kernel regime with feature learning.
Advanced18 mincodedemoafter Overparameterization & Generalization (Double Descent) - 05Tree Search Reasoning: Allocating Inference Budget Across Prefixes
Tree search spends inference budget on partial reasoning prefixes, using local verifier scores, frontier expansion, and max backups to decide which branches deserve more thought.
Advanced32 mincodedemoafter Test-Time Compute: Spending Inference Budget on Search, Process Reward Models: Step-Level Verifiers for Reasoning, Decoding & Sampling: Temperature, Top-p & Inference-Time Control
All notebooks
Every published notebook in this domain.
Overparameterization & Generalization (Double Descent)
Test error can peak at the interpolation threshold, where a model first fits its training data exactly, and fall again as the model grows past it.
Scaling Laws & Emergent Abilities
Empirical power laws for how pretraining loss falls with parameters, data and compute, how to split a fixed budget between model size and tokens, and why some abilities seem to appear suddenly.
Test-Time Compute: Spending Inference Budget on Search
Test-time compute spends extra inference budget on sampling, verification, and selection; it helps when the generator can produce good candidates and the verifier ranks them reliably.
Neural Tangent Kernel (NTK) & Infinite-Width Limits
In the infinite-width limit under the NTK parameterization, squared-loss training by gradient flow or with a sufficiently small learning rate follows the network's linearization at initialization. When the training kernel is invertible, the converged predictions add a kernel-regression fit of the initial errors to the initial predictions. The notebook contrasts this fixed-kernel regime with feature learning.
Tree Search Reasoning: Allocating Inference Budget Across Prefixes
Tree search spends inference budget on partial reasoning prefixes, using local verifier scores, frontier expansion, and max backups to decide which branches deserve more thought.