A loss landscape is the loss viewed as a function of the parameters. A minimum is sharp when the loss rises quickly as the parameters move away from it in some direction, and flat when it rises slowly in every direction. Keskar et al. (2017) found that large-batch training reached sharper minima and generalized worse than small-batch SGD in their experiments.
A newer notebook covers this concept. This older page keeps its summary, math and sources.
Concept 4 of 100SharpnessOptimization · Study phase 3: Optimization & generalization
Question
Sharp and flat minima are usually drawn in two dimensions; what does sharpness mean in millions of dimensions, where the curvature differs greatly from one direction to another?
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you move a control in the demo, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the demo shows with the key equation and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueContinue in the newer notebook.
The newer notebook covers the same concept and has an interactive demo.
Why It Matters for Modern Models
A loss landscape is the loss viewed as a function of the parameters. A minimum is sharp when the loss rises quickly as the parameters move away from it in some direction, and flat when it rises slowly in every direction. Keskar et al. (2017) found that large-batch training reached sharper minima and generalized worse than small-batch SGD in their experiments.
Sharpness alone does not decide generalization, because rescaling a network can make a flat minimum sharp without changing the function it computes (Dinh et al., 2017).
SAM improved test accuracy on image-classification benchmarks and when fine-tuning pretrained models, at roughly twice the cost per step (Foret et al., 2021).
Questions to Think About
How does mode connectivity (separate minima joined by low-loss paths) relate to flatness and to weight averaging?
Interactive Demos
Core Math
If you prefer intuition first, try the demos above, then come back for the math.
Key Equation
wmin∥ϵ∥p≤ρmaxL(w+ϵ)
Sharpness-aware minimization (SAM) objective:
wmin∥ϵ∥p≤ρmaxL(w+ϵ)
In practice (for p=2):
Approximate the worst-case perturbation with one gradient step: ϵ(w)≈ρ∥∇L(w)∥2∇L(w)
Update using the gradient at the perturbed weights: ∇L(w+ϵ(w))
Each SAM step therefore needs two gradient computations.
On the theory side, Kawaguchi (2016) proved that every local minimum of a deep linear network is a global minimum, and extended a similar result to nonlinear networks under strong simplifying assumptions.
Beyond interpolation threshold: where Sharpness breaks down, taken up in Double Descent. Open Double Descent
Classical intuition says that more parameters means more overfitting. Double descent shows test error falling again past the point where a model fits its training data exactly (Belkin et al., 2019). One proposed explanation is that larger models find flatter minima, though the link between flatness and generalization is debated (Dinh et al., 2017).
Edge of Stability: where Adam breaks down, taken up on this page. Open Adam
With full-batch gradient descent the sharpness, the largest eigenvalue of the Hessian, rises until it hovers near 2/η, where η is the step size; the loss keeps falling, though not monotonically, even though classical analysis says steps this large should diverge (Cohen et al., 2021). Adam behaves the same way for its preconditioned sharpness, which settles near 38/η when β₁ = 0.9 (Cohen et al., 2022).
Sharp minima: where Efficiency breaks down, taken up on this page. Open Efficiency
Quantization and low-rank approximation perturb the weights (and, for quantized activations, the activations). Where the loss is sharp, small perturbations cause large increases in loss, so compression can degrade quality much more than its size suggests.
Linear patching misses curvature: where Circuit Discovery breaks down, taken up on this page. Open Circuit Discovery
Fast circuit-finding methods such as attribution patching estimate the effect of an intervention with a first-order Taylor expansion. Where the model responds in a strongly nonlinear way to the patched activations, as with large interventions or saturated attention, the linear estimate can be badly wrong and can point to the wrong circuit.
Where else does the same idea appear?
Geometry ↔ Generalization: an analogy between Sharpness and Double Descent. Open Double Descent
Both study the shape of optimization: loss landscapes focus on local geometry (sharpness), double-descent on global capacity (interpolation threshold).
Saddle point game: an analogy between Sharpness and GANs. Open GANs
GAN training is a min-max game whose solutions are saddle points rather than minima. Gradient methods can circle around such points instead of converging to them, one source of the oscillation seen in GAN training.
Which ideas are its mathematical duals?
Flatness ↔ Complexity: Learning Theory is its mathematical dual. Open Learning Theory
Flat minima correspond to larger volumes of parameter space that implement similar functions, which often aligns with lower effective complexity in PAC-Bayes and MDL-style arguments. The link is debated: sharpness can be changed by reparameterizing a network without changing the function it computes (Dinh et al., 2017).