Domain Neighborhood
Scaling
How loss and capability change with parameters, data, and compute; how to allocate a training budget; and why some abilities appear suddenly at scale.
Start here. Predict once, then carry the invariant forward.
NDClosspower lawLearner lensWhat makes this domain feel navigable?
Take this lensStabilize the first mechanism, make one prediction, then move one node forward.
Recommended Route
Start here, then follow the prerequisites forward.
This sequence is ordered for learning rather than inventory. Published notebooks with an unavailable prerequisite—or a same-domain route step that depends on one—are labeled in the full inventory instead of being presented as ready steps.
- 01Overparameterization & Generalization (Double Descent)
Test error can peak at the interpolation threshold then fall again as models get larger: why modern overparameterized nets still generalize.
16 mincodedemoafter Loss Landscapes, Sharpness & Flat MinimaCheck Loss Landscapes, Sharpness & Flat Minima first if the symbols feel slippery.
- 02Scaling Laws & Emergent Abilities
Empirical power laws that predict how loss and capability improve with parameters, data, and compute, and how to choose compute-optimal training runs.
18 mincodedemoafter Scaled Dot-Product Attention & Transformer Layers, Overparameterization & Generalization (Double Descent)Why this follows: Scaling Laws & Emergent Abilities uses Overparameterization & Generalization (Double Descent) directly.
- 03Neural Tangent Kernel (NTK) & Infinite-Width Limits
A limit where wide neural networks behave like kernel methods: the model linearizes around initialization and training becomes kernel regression.
18 mincodedemoafter Overparameterization & Generalization (Double Descent)Why this follows: both pages keep the scaling thread active.
- 04Test-Time Compute: Spending Inference Budget on Search
Test-time compute spends extra inference budget on sampling, verification, and selection; it helps when the generator can produce good candidates and the verifier ranks them reliably.
22 mincodedemoafter Scaling Laws & Emergent Abilities, Process Reward Models: Step-Level Verifiers for Reasoning, Reward Hacking: Overoptimizing Preference ProxiesWhy this follows: both pages keep the scaling thread active.
- 05Tree Search Reasoning: Allocating Inference Budget Across Prefixes
Tree search spends inference budget on partial reasoning prefixes, using local verifier scores, frontier expansion, and max backups to decide which branches deserve more thought.
24 mincodedemoafter Test-Time Compute: Spending Inference Budget on Search, Process Reward Models: Step-Level Verifiers for Reasoning, Decoding & Sampling: Temperature, Top-p & Inference-Time ControlWhy this follows: Tree Search Reasoning: Allocating Inference Budget Across Prefixes uses Test-Time Compute: Spending Inference Budget on Search directly.
All Published Notebooks
Browse the territory.
Overparameterization & Generalization (Double Descent)
Test error can peak at the interpolation threshold then fall again as models get larger: why modern overparameterized nets still generalize.
Scaling Laws & Emergent Abilities
Empirical power laws that predict how loss and capability improve with parameters, data, and compute, and how to choose compute-optimal training runs.
Neural Tangent Kernel (NTK) & Infinite-Width Limits
A limit where wide neural networks behave like kernel methods: the model linearizes around initialization and training becomes kernel regression.
Test-Time Compute: Spending Inference Budget on Search
Test-time compute spends extra inference budget on sampling, verification, and selection; it helps when the generator can produce good candidates and the verifier ranks them reliably.
Tree Search Reasoning: Allocating Inference Budget Across Prefixes
Tree search spends inference budget on partial reasoning prefixes, using local verifier scores, frontier expansion, and max backups to decide which branches deserve more thought.
Advanced Bridges