Double descent is a pattern in which test error falls, rises to a peak near the model size that first fits the training data exactly, then falls again as the model grows. Image classifiers that can fit even randomly labeled training data still generalize well on real labels, which the classical U-shaped picture of test error does not explain (Zhang et al., 2017).
A newer notebook covers this concept. This older page keeps its summary, math and sources.
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you move a control in the demo, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the demo shows with the key equation and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueContinue in the newer notebook.
The newer notebook covers the same concept and has an interactive demo.
Why It Matters for Modern Models
Double descent is a pattern in which test error falls, rises to a peak near the model size that first fits the training data exactly, then falls again as the model grows. Image classifiers that can fit even randomly labeled training data still generalize well on real labels, which the classical U-shaped picture of test error does not explain (Zhang et al., 2017).
Large language models are a different case: they are usually trained on more tokens than they have parameters (Chinchilla used about 20 tokens per parameter), and compute-optimal scaling laws model how their test loss depends on both model size and data (Hoffmann et al., 2022).
Questions to Think About
Among the infinitely many solutions that fit the training data, which one do SGD and Adam select, and why does that choice matter for test error?
Interactive Demos
Core Math
If you prefer intuition first, try the demos above, then come back for the math.
Key Equation
E[(y−y^)2]=bias2+variance+σ2
Classical learning theory predicts that test error is a U-shaped function of model capacity. Many models instead show double descent: test error peaks near the interpolation threshold, where the model first fits the training data exactly, and falls again as capacity keeps growing (Belkin et al., 2019; Nakkiran et al., 2020).
For squared loss, the expected test error splits into three parts:
E[(y−y^)2]=bias2+variance+σ2
In minimum-norm least squares, the variance term blows up when the number of parameters is close to the number of training examples and falls again beyond it, because among the many interpolating solutions the algorithm picks the one with the smallest norm (Hastie et al., 2022).
Key Papers
Understanding Deep Learning Requires Rethinking Generalization
Small data: where Scaling Laws breaks down, taken up on this page. Open Scaling Laws
Scaling-law fits assume enough data that models do not simply memorize their training set. Near the size at which a model can just fit its training data exactly, test error can rise before falling again as size grows (Nakkiran et al., 2020), which a smooth power law does not capture.
Bias-variance puzzle: where Learning Theory breaks down, taken up on this page. Open Learning Theory
Classical bias-variance reasoning says that test error rises once a model is complex enough to fit the noise in its training data. Double descent adds a second fall: past the point where the model fits the training data exactly, larger models generalize better again (Belkin et al., 2019). Explaining this needed new theory, such as benign overfitting (Bartlett et al., 2020).
Beyond interpolation threshold: where Sharpness breaks down, taken up on this page. Open Sharpness
Classical intuition says that more parameters means more overfitting. Double descent shows test error falling again past the point where a model fits its training data exactly (Belkin et al., 2019). One proposed explanation is that larger models find flatter minima, though the link between flatness and generalization is debated (Dinh et al., 2017).
Fixed features suffice: where NTK breaks down, taken up on this page. Open NTK
Double descent does not need feature learning: it already appears in random-feature models, whose features are fixed (Belkin et al., 2019), and so in the lazy regime that NTK describes. What the NTK picture misses is the learning of new features, which deep networks also do.
Where else does the same idea appear?
Geometry ↔ Generalization: an analogy between Double Descent and Sharpness. Open Sharpness
Both study the shape of optimization: loss landscapes focus on local geometry (sharpness), double-descent on global capacity (interpolation threshold).