Foundation Lab

Loss Landscapes, Sharpness & Flat Minima

Frontier models rely on implicit flat-minima bias (mini-batch SGD, data augmentation, weight decay) for generalization

Concept 4 of 100OptimizationPhase 3

#4SharpnessOptimization

key equation\min_w \max_{\|\epsilon\|_p \le \rho} L(w + \epsilon)

Selected Foundation Object

Keep the equation fixed; move through the evidence.

Concept 4 of 100SharpnessOptimization / Phase 3: Optimization & generalization

Current question

Most expositions show "sharp vs flat" in 2D, but we lack intuitive stories for high-dimensional anisotropic sharpness

\min_w \max_{\|\epsilon\|_p \le \rho} L(w + \epsilon)

PredictionCommit before the demo.

Ask what should change when the equation is manipulated, then let the visualization test that expectation.

EvidenceCompare local witness and source.

Use the runnable panel, the key equation, and canonical papers as separate forms of evidence for the same object.

InvariantName what survives notation changes.

The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.

Next moveContinue in the newer notebook.

This atlas page keeps the working demo; the domain notebook carries the fuller Intuition -> Math -> Code -> Demo sequence.

Frontier models rely on implicit flat-minima bias (mini-batch SGD, data augmentation, weight decay) for generalization
Fine-tuning and RLHF pipelines sometimes adopt SAM-like ideas to stabilize training

What is still poorly explained in textbooks and papers:

Most expositions show "sharp vs flat" in 2D, but we lack intuitive stories for high-dimensional anisotropic sharpness
How mode connectivity (many minima connected by low-loss paths) interacts with flatness and weight averaging

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation

\min_w \max_{\|\epsilon\|_p \le \rho} L(w + \epsilon)

SAM objective:

\min_w \max_{\|\epsilon\|_p \le \rho} L(w + \epsilon)

In practice:

Take a single gradient step to find "worst-case" perturbation: $\epsilon(w) \approx \rho \frac{\nabla L(w)}{\|\nabla L(w)\|_2}$
Update using gradient at the perturbed weights: $\nabla L(w+\epsilon(w))$