Scaling laws are used to choose model size and training tokens for a compute budget: Chinchilla (70B parameters, 1.4T tokens) used the same compute as the 280B-parameter Gopher and outperformed it (Hoffmann et al., 2022).
A newer notebook covers this concept. This older page keeps its summary, math and sources.
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you move a control in the demo, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the demo shows with the key equation and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueContinue in the newer notebook.
The newer notebook covers the same concept and has an interactive demo.
Why It Matters for Modern Models
Scaling laws are used to choose model size and training tokens for a compute budget: Chinchilla (70B parameters, 1.4T tokens) used the same compute as the 280B-parameter Gopher and outperformed it (Hoffmann et al., 2022).
Similar power-law fits have been reported for generative models of images, video and multimodal data (Henighan et al., 2020).
Questions to Think About
How does performance on individual tasks change with scale, and how much of an apparent jump comes from the choice of metric?
Interactive Demos
Core Math
If you prefer intuition first, try the demos above, then come back for the math.
Key Equation
L(N,D)≈E+NαA+DβB
Test loss L follows approximate power laws in the number of parameters N and training tokens D (Hoffmann et al., 2022):
L(N,D)≈E+NαA+DβB
where E is the irreducible loss and A,B,α,β are fitted constants. Training compute is approximately C≈6ND floating-point operations.
Compute-optimal allocation: for a fixed budget C, Hoffmann et al. found that the best N and D both grow roughly as C0.5, so D∝N at about 20 tokens per parameter. Doubling compute therefore multiplies each of them by about 2≈1.4. Kaplan et al. (2020) had estimated that parameters should grow much faster than data; Hoffmann et al. attribute much of the difference to Kaplan et al. using one fixed learning-rate schedule and number of training tokens for all model sizes.
Emergent abilities: Wei et al. (2022) reported abilities, such as multi-digit arithmetic, that appear abruptly above some model scale. Schaeffer et al. (2023) argue that much of this abruptness comes from discontinuous metrics such as exact-match accuracy, and that smoother metrics show gradual improvement.
Which limitations does it address, or leave to a later idea?
More params, same FLOPs: a limitation of Scaling Laws that MoE addresses. Open MoE
In a dense model every parameter is used for every token, so parameter count and compute per token grow together. A mixture-of-experts model adds parameters while keeping the compute per token roughly fixed, by sending each token to only a few experts (Shazeer et al., 2017; Fedus et al., 2022).
Where does it break down?
Small data: where Scaling Laws breaks down, taken up in Double Descent. Open Double Descent
Scaling-law fits assume enough data that models do not simply memorize their training set. Near the size at which a model can just fit its training data exactly, test error can rise before falling again as size grows (Nakkiran et al., 2020), which a smooth power law does not capture.
Token units shift: where Scaling Laws breaks down, taken up in Tokenization. Open Tokenization
Scaling laws report cross-entropy per token, but tokenizer changes shift what "a token" means. Curves can look better/worse unless normalized (bits-per-byte).
Where else does the same idea appear?
Budget allocation: an analogy between Scaling Laws and LLM Serving. Open LLM Serving
Scaling laws ask how to divide a compute budget between model size and data for the lowest loss. Serving asks how to divide GPU memory and compute among users to serve the most requests within latency targets. Both are optimization under a fixed budget.
Compression efficiency: an analogy between Scaling Laws and Tokenization. Open Tokenization
Vocabulary size changes how many tokens a text takes and how many embedding parameters a model has. Tao et al. (2024) find that the compute-optimal vocabulary grows with model size; they estimate that Llama 2 70B would have been better served by a vocabulary of at least 216,000 tokens than by its 32,000.
Which ideas are its mathematical duals?
Width ↔ Depth: NTK is its mathematical dual. Open NTK
NTK theory describes networks in the limit of infinite width. Scaling laws are empirical: Kaplan et al. (2020) found that, at a fixed number of parameters, loss depends only weakly on the balance between width and depth. Both ask how the shape of a network matters at a given size.