Foundation Lab

SGD & Momentum: The Workhorses of Optimization

Momentum is still used to train most vision models and is the default for many frameworks

Concept 57 of 100OptimizationPhase 3

#57SGD+MomentumOptimization

key equation

v_{t+1} = \mu v_t + \nabla L, \quad \theta_{t+1} = \theta_t - \eta v_{t+1}

Selected Foundation Object

Keep the equation fixed; move through the evidence.

Concept 57 of 100SGD+MomentumOptimization / Phase 3: Optimization & generalization

Current question

Momentum = exponential moving average of gradients, so it smooths over mini-batch noise

v_{t+1} = \mu v_t + \nabla L, \quad \theta_{t+1} = \theta_t - \eta v_{t+1}

PredictionCommit before tracing the equation.

Ask what should change under a concrete input, then trace that expectation through the equation.

EvidenceCompare the equation and source.

Use the key equation and canonical papers as the available witnesses, without implying that a runnable panel exists.

InvariantName what survives notation changes.

The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.

Next moveContinue through the atlas.

Use prerequisites, dependents, and semantic links to repair the next gap without leaving the object behind.

Momentum is still used to train most vision models and is the default for many frameworks
Understanding momentum explains why adaptive methods (Adam) can be worse for generalization
Nesterov momentum provides optimal convergence rates for convex optimization

What is still poorly explained in textbooks and papers:

Momentum = exponential moving average of gradients, so it smooths over mini-batch noise
Heavy ball analogy: momentum lets you roll through small bumps and ravines in the loss landscape
Nesterov looks ahead: "if I keep going this way, what would the gradient be?"

If you want intuition first, start with the key equation and cited sources. Come back here for the full walkthrough.

Key Equation

v_{t+1} = \mu v_t + \nabla L, \quad \theta_{t+1} = \theta_t - \eta v_{t+1}

Vanilla SGD:

\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)

Momentum (Polyak):

v_{t+1} = \mu v_t + \nabla L(\theta_t)

\theta_{t+1} = \theta_t - \eta v_{t+1}

Nesterov Momentum (look-ahead gradient):

v_{t+1} = \mu v_t + \nabla L(\theta_t - \eta \mu v_t)

\theta_{t+1} = \theta_t - \eta v_{t+1}

With momentum $\mu \approx 0.9$ , effective learning rate is $\eta / (1 - \mu) = 10\eta$ .

Sutskever et al.2013ICML

Nesterov1983Soviet Mathematics Doklady

Choose the next question to carry this object forward.