Continuous Function

Loss Landscapes, Sharpness & Flat Minima

What 2D slices of a loss surface show and hide, how Hessian curvature measures sharpness, why the link between flat minima and generalization is contested, and how SAM trains toward flat regions.

Intermediate · Undergraduate mathematics · about 16 minutes

Reading map and next steps

Intuition

The idea in plain words, before the symbols.

Training moves the parameters downhill on the loss. For a neural network that surface has one dimension per parameter: millions or billions of them.

A picture of a loss landscape is therefore a slice: the loss along one or two chosen directions through parameter space, with everything else held fixed (Li et al., 2018). Slices are useful, but what they show depends on which directions are chosen and how they are scaled.

Two terms come up often:

  • Sharp minimum: a small change to the weights raises the loss a lot. The curvature, measured by the largest eigenvalues of the Hessian, is high.
  • Flat minimum: the loss stays low over a wide region around the minimum.

The idea that flat minima generalize better goes back to Hochreiter and Schmidhuber (1997), and Keskar et al. (2017) found that large-batch training tends to converge to sharper minima that generalize worse. The idea is contested. Dinh et al. (2017) showed that rescaling the weights of a ReLU network can make a minimum arbitrarily sharp without changing the function the network computes, so sharpness measured this way cannot by itself explain generalization.

Sharpness-aware minimization (SAM; Foret et al., 2021) builds a preference for flat regions into training: it minimizes the highest loss within a small ball around the current weights, not just the loss at one point.

Study prompts for this section: Intuition
Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Mathematics

Definitions, assumptions and the derivation.

Equation 1Equation 2

Let L(w)L(w) be the training loss for parameters ww. Near a point ww, a quadratic approximation is:

L(w+ϵ)≈L(w)+ϵ⊤∇L(w)+12 ϵ⊤H(w) ϵ,L(w+\epsilon) \approx L(w) + \epsilon^\top \nabla L(w) + \tfrac12\,\epsilon^\top H(w)\,\epsilon,

where H(w)=∇2L(w)H(w) = \nabla^2 L(w) is the Hessian (curvature).

A common proxy for "sharpness" is how much loss can increase under a small perturbation. SAM turns that local-neighborhood idea into an objective:

Δρ(w):=max⁡∥ϵ∥2≤ρ(L(w+ϵ)−L(w)),LSAM(w):=max⁡∥ϵ∥2≤ρL(w+ϵ),min⁡wLSAM(w),ϵ^(w)≈ρ ∇L(w)∥∇L(w)∥2.\begin{aligned} \Delta_{\rho}(w) &:= \max_{\|\epsilon\|_2 \le \rho} \big(L(w+\epsilon) - L(w)\big),\\ L_{\mathrm{SAM}}(w) &:= \max_{\|\epsilon\|_2 \le \rho} L(w+\epsilon), \qquad \min_w L_{\mathrm{SAM}}(w),\\ \hat{\epsilon}(w) &\approx \rho\,\frac{\nabla L(w)}{\|\nabla L(w)\|_2}. \end{aligned}

Here Δρ\Delta_\rho is the largest increase in loss within distance ρ\rho. At a minimum, where ∇L(w)=0\nabla L(w)=0 and H(w)H(w) has no negative eigenvalues, the quadratic approximation gives Δρ(w)≈12ρ2λmax⁡(H)\Delta_\rho(w)\approx\tfrac12\rho^2\lambda_{\max}(H): sharpness is the largest Hessian eigenvalue, scaled by the radius.

SAM minimizes the worst-case training loss in the ρ\rho-ball. In practice it approximates the inner maximization with one normalized gradient step, ϵ^(w)\hat\epsilon(w) above, and then updates the weights with the gradient taken at the perturbed point: w←w−η ∇L(w+ϵ^(w))w\leftarrow w-\eta\,\nabla L\big(w+\hat\epsilon(w)\big).

The same λmax⁡\lambda_{\max} limits gradient descent: on a quadratic, a fixed step converges only if η<2/λmax⁡\eta<2/\lambda_{\max} (see Gradient Descent). Cohen et al. (2021) found that full-batch gradient descent on neural networks pushes λmax⁡\lambda_{\max} up until it hovers just above 2/η2/\eta, while the loss keeps falling over the long run. The code below checks the quadratic bound.

Study prompts for this section: Mathematics
Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Code

The same calculation as runnable code, in the notation of the derivation.

Code example 1import numpy as np # Anisotropic quadratic: sharp along x (large curvature), flat along y (sm...python
import numpy as np

# Anisotropic quadratic: sharp along x (large curvature), flat along y (small curvature).
a, b = 20.0, 1.0  # Hessian eigenvalues
lam_max = max(a, b)

def L(w): return 0.5 * (a * w[0] ** 2 + b * w[1] ** 2)
def grad(w): return np.array([a * w[0], b * w[1]])

print("stable if eta < 2/lambda_max =", round(2.0 / lam_max, 3))

for eta in [0.02, 0.08, 0.12]:
    w = np.array([1.0, 1.0])
    ok = True
    for _ in range(60):
        w = w - eta * grad(w)
        if not np.isfinite(L(w)) or np.linalg.norm(w) > 1e6:
            ok = False; break
    print("eta =", eta, "final L =", round(L(w), 6), "status =", "ok" if ok else "diverged")
Study prompts for this section: Code
Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Demo

Change an input, predict the result, and see which quantities respond.

ExploreChange one control at a time and name what responds.An optional observation guide follows the demo.

Explore Loss Landscapes, Sharpness & Flat Minima

Loading the interactive demo…

Observation guide

Optional and ungraded. Opening a guide saves nothing.

Choose what to inspect in Loss Landscapes, Sharpness & Flat Minima, then open its guide.

The demo has two parts.

  • Part 1: a two-parameter toy loss with a sharp and a flat minimum. Step SGD and SAM from the same start, read the largest Hessian eigenvalue λmax⁡\lambda_{\max} from the heat map, and predict which path ends in the less sharp region.
  • Part 2: a toy sharpness trace drawn against the line 2/η2/\eta. Predict whether it stays below the line, hovers near it, or crosses it and diverges.

Both parts use made-up toy surfaces and traces, not measurements from a trained network.

Study prompts for this section: Demo
Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

In focus

Concept: Loss Landscapes, Sharpness & Flat Minima

What is the smallest example of Loss Landscapes, Sharpness & Flat Minima you can work through by hand, and what does it show?

BeforeAdam OptimizerNextGradient Clipping & Explosion Prevention
DetailsOptimization
About this itemOptimization
6 sources citedChecking this browser
Reference idconcept:optimization/loss-landscapes
Math2 equationsL(w+\epsilon) \approx L(w) + \epsilon^T \nabla L(w) + \tfrac12 \epsilon^T H(w) \epsilon,CodeLoss Landscapes, Sharpness & Flat Minima code example 1a, b = 20.0, 1.0 # Hessian eigenvaluesDemoLoss Landscapes, Sharpness & Flat Minima interactive demoChange one control at a time and name what responds.

After the first pass

Go deeper

Optional material for a second reading. Sources are always shown; the other panels open from their headings or from these links, and closing one keeps what you wrote or revealed.
What to look for in each sectionChoose a section and what you expect to see in it, then read short advice.

Section by section

Choose what to look for in each section

What 2D slices of a loss surface show and hide, how Hessian curvature measures sharpness, why the link between flat minima and generalization is contested, and how SAM trains toward flat regions.

Choose what to look for (no demo yet)01 / Intuition
Illustration of contour loss basins, sharp and flat minima, and descent trajectories.
What to look for

Start with the picture, metaphor, or geometric mechanism.

Choose first

Choose what you expect to see in this section of Loss Landscapes, Sharpness & Flat Minima.

Questions to ask of the pictureChoose a part of the picture and what you expect, then read short advice.

Looking at the picture

Questions to ask of the picture

What 2D slices of a loss surface show and hide, how Hessian curvature measures sharpness, why the link between flat minima and generalization is contested, and how SAM trains toward flat regions.

4 of 4 sections writtenNo live demo yet
Question

Which part of the picture should you look at first?

Choose first

Pick the part of the picture you expect to explain Loss Landscapes, Sharpness & Flat Minima.

Sources

References for this page

A reference shows where an idea comes from; it does not vouch for every step on this page.
paper · 2018Visualizing the Loss Landscape of Neural NetsLi et al.
Why it is cited

Plots 1D and 2D slices of neural-network loss surfaces and introduces filter normalization so that slices of different networks can be compared (NeurIPS 2018).

Open source
paper · 2017On Large-Batch Training for Deep Learning: Generalization Gap and Sharp MinimaKeskar et al.
Why it is cited

Finds that large-batch training tends to converge to sharper minima that generalize worse, measuring sharpness as the loss increase within a small neighborhood (ICLR 2017).

Open source
paper · 2021Sharpness-Aware Minimization for Efficiently Improving GeneralizationForet et al.
Why it is cited

Introduces SAM, which minimizes the worst-case training loss in a small neighborhood of the weights using a one-step approximation of the inner maximization (ICLR 2021).

Open source
paper · 1997Flat MinimaHochreiter and Schmidhuber
Why it is cited

Early argument that minima where the loss stays low over a large region of weight space generalize better. Neural Computation 9(1), 1-42.

Open source
paper · 2017Sharp Minima Can Generalize For Deep NetsDinh et al.
Why it is cited

Shows that the parameter symmetries of ReLU networks allow equivalent networks at arbitrarily sharper minima, so common flatness measures cannot by themselves explain generalization (ICML 2017).

Open source
paper · 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of StabilityCohen et al.
Why it is cited

Finds that during full-batch gradient descent the largest Hessian eigenvalue rises until it hovers just above 2/η, while the loss decreases over long timescales but not monotonically (ICLR 2021).

Open source
What each claim rests on1 claim, each with its sources, where it appears on this page, and its caveats.

Claims and sources

What 2D slices of a loss surface show and hide, how Hessian curvature measures sharpness, why the link between flat minima and generalization is contested, and how SAM trains toward flat regions.

Checked against the cited passages

The checks were made by this site, not by an independent reviewer.

Loss-landscape visualizations are 1D/2D diagnostic slices through high-dimensional parameter space, not the full neural-network surface; sharpness/flatness depend on perturbation scale and network symmetries, and SAM minimizes worst-case training loss in a perturbation neighborhood.
Sources
Visualizing the Loss Landscape of Neural NetsPlots 1D and 2D slices of neural-network loss surfaces and introduces filter normalization so that slices of different networks can be compared (Ne...On Large-Batch Training for Deep Learning: Generalization Gap and Sharp MinimaFinds that large-batch training tends to converge to sharper minima that generalize worse, measuring sharpness as the loss increase within a small...Sharpness-Aware Minimization for Efficiently Improving GeneralizationIntroduces SAM, which minimizes the worst-case training loss in a small neighborhood of the weights using a one-step approximation of the inner max...
What the sources support

Li supports 1D/2D slices and filter-wise normalization caveats. Keskar supports sharp/flat minima as perturbation-sensitivity diagnostics tied to generalization, while using an imperfect epsilon-scale-depend...

Where to see it on this page
Visualizing the Loss Landscape of Neural NetsOn Large-Batch Training for Deep Learning: Generalization Gap and Sharp MinimaSharpness-Aware Minimization for Efficiently Improving GeneralizationEquation 1Equation 2Demo
Equation 1
L(w+ϵ)≈L(w)+ϵ⊤∇L(w)+12 ϵ⊤H(w) ϵ,L(w+\epsilon) \approx L(w) + \epsilon^\top \nabla L(w) + \tfrac12\,\epsilon^\top H(w)\,\epsilon,
Equation 2
Δρ(w):=max⁡∥ϵ∥2≤ρ(L(w+ϵ)−L(w)),LSAM(w):=max⁡∥ϵ∥2≤ρL(w+ϵ),min⁡wLSAM(w),ϵ^(w)≈ρ ∇L(w)∥∇L(w)∥2.\begin{aligned} \Delta_{\rho}(w) &:= \max_{\|\epsilon\|_2 \le \rho} \big(L(w+\epsilon) - L(w)\big),\\ L_{\mathrm{SAM}}(w) &:= \max_{\|\epsilon\|_2 \le \rho} L(w+\epsilon), \qquad \min_w L_{\mathrm{SAM}}(w),\\ \hat{\epsilon}(w) &\approx \rho\,\frac{\nabla L(w)}{\|\nabla L(w)\|_2}. \end{aligned}
Caveat

Does not cover the edge of stability or the 2/eta code and demo (see Cohen et al.), Hessian spectra of real networks, any guarantee that flat minima generalize, or a sharpness measure unchanged by reparamete...

StatusChecked against the cited passages.

Li supports low-dimensional 1D/2D slices and filter-normalized comparisons. Keskar supports sharp/flat minima as perturbation-sensitivity diagnostics tied to generalization but with an imperfect epsilon-dependent metric. Foret supports SAM's worst-case neighborhood objective and first-order perturbation approximation. Not checked: the edge of stability, the 2/eta code, part 2 of the demo, real-network Hessian spectra, any guarantee that flat minima generalize, and reparameterization-invariant sharpness.

Explain it without the pageWrite your own explanation before asking for feedback. Your draft and any hints stay while this page is open.

Practice · Loss Landscapes, Sharpness & Flat Minima

Try the idea in your own words

What 2D slices of a loss surface show and hide, how Hessian curvature measures sharpness, why the link between flat minima and generalization is contested, and how SAM trains toward flat regions.

Concept · Selected for practice

Loss Landscapes, Sharpness & Flat Minima

What it rests on: Sources: Visualizing the Loss Landscape of Neural Nets; On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima; Sharpness-A...

Context and links

Optimization

Choose a task

Explain the mechanism

For Loss Landscapes, Sharpness & Flat Minima: What is the smallest example of Loss Landscapes, Sharpness & Flat Minima you can work through by hand, and what does it show? Explain your answer, including what changes, why, and which assumption matters.

No answer yet

A rough first thought is enough. Your draft stays when you change tasks.

Your draft stays on this page and is cleared when you leave or reload.

A little help · Explain

Open one hint at a time. These are suggestions, not your answer or a grade.

0 of 3 hints shown for this question.

    Clearing your answer keeps this help record. Outside help is your own report; this page cannot check it.

    Where am I stuck? (optional)
    Your own description, not an automatic diagnosis

    Choose one, or leave this unspecified. Select it again to clear it.

    Take your draft to a feedback conversation

    No AI feedback runs here. You can copy a prompt to use elsewhere; nothing is sent automatically. Review the text before sharing, and leave out private information.

    Write an attempt before copying a feedback prompt.

    A draft, or an AI reply to it, is not a test of what you have learned. To check that, try a different case later without help.

    Ask about this pagePick one item on this page and copy a prompt about it into an AI assistant you already use. Notes you write here stay in this browser.
    Ask about this pageClose
    ConceptLoss Landscapes, Sharpness & Flat MinimaSources: Visualizing the Loss Landscape of Neural Nets; On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima; Sharpness-A...a, b = 20.0, 1.0 # Hessian eigenvalues

    Your question

    Choose what your question is about

    Pick the idea, equation, source, code, claim, misconception or demo state you want to ask about. The prompt you copy includes it, so the answer can stay on that item.
    Next local actionNo local draft saved yet

    Open the draft below to save one note and next action in this browser.

    conceptOptimization

    Loss Landscapes, Sharpness & Flat Minima

    Starting question

    What is the smallest example of Loss Landscapes, Sharpness & Flat Minima you can work through by hand, and what does it show?

    SourcesCheck the 6 cited sources listed under Sources.Notes can be saved for this item
    Points of view on this item

    Fixed prompts generated from this item. They are not comments from people or independent reviews.

    Learner’s first stepAsk what would make "Loss Landscapes, Sharpness & Flat Minima" feel predictable rather than familiar.
    Assumption

    The 6 cited sources must support this exact item, not just the surrounding topic.

    Source-checking summary

    Connect the definition to one equation, piece of code or demo before widening the discussion.

    Proposed experiment

    Change one input, then check that the same relationship holds in the mathematics, the code and the demo.

    Next action

    You can state the mechanism in your own words

    Evidence4 checks
    ObservationChecking for a saved observation
    ActionReady for one action
    PromptLearner prompt ready to copy
    Go to this item
    01ObservationChecking what this browser has saved
    02EvidenceChecking for a saved observation
    03SourcesCheck the 6 cited sources listed under Sources.
    04Next stepSave one next action
    Local action draftNo local draft saved yetOpen when you are ready to write one next action
    Local action draft

    This draft stays in this browser and is attached to this item only.

    No local draft saved.
    Evidence to inspect
    • What the 6 cited sources say about this exact item
    • The definition, its prerequisites, and a contrasting concept
    • The equation or code that makes the concept concrete
    • One demo state that shows what stays the same as the inputs change
    What would resolve this
    • You can state the mechanism in your own words
    • You can name the prerequisite that would clear up the confusion
    • You can predict how the result changes when one input changes
    Reference idconcept/concept-notebook/optimization/loss-landscapes concept:optimization/loss-landscapes