Continuous Function

Foundation reference

Stochastic Gradient Descent & Momentum

Stochastic gradient descent (SGD) steps against the gradient of the loss on a mini-batch; momentum steps against an exponentially weighted sum of past mini-batch gradients. SGD with momentum was the standard optimizer for image classifiers such as ResNet, which was trained with momentum 0.9 (He et al., 2016).

Concept 57 of 100OptimizationStudy phase 3
Open the demoAll concepts
Key equation
vt+1=μvt+∇L,θt+1=θt−ηvt+1v_{t+1} = \mu v_t + \nabla L, \quad \theta_{t+1} = \theta_t - \eta v_{t+1}
Reading map and next steps

Study guide

Where to start

Concept 57 of 100SGD & MomentumOptimization · Study phase 3: Optimization & generalization
Question

How does the velocity, an exponentially weighted sum of past gradients, average out mini-batch noise while adding up a direction that stays the same from step to step?

Open the demoGo to the mathAll conceptsNext concept
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.

Before you move a control in the demo, say what you expect to change.

CheckCheck against the math and the papers.

Compare what the demo shows with the key equation and with the papers cited below.

GeneralizeName what stays the same.

Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.

ContinueFollow the connections.

Use the prerequisites and related concepts at the end of the page to fill the next gap.

Why It Matters for Modern Models

  • Stochastic gradient descent (SGD) steps against the gradient of the loss on a mini-batch; momentum steps against an exponentially weighted sum of past mini-batch gradients. SGD with momentum was the standard optimizer for image classifiers such as ResNet, which was trained with momentum 0.9 (He et al., 2016).
  • Adaptive methods such as Adam often reduce the training loss faster, but Wilson et al. (2017) found that on several benchmarks their solutions generalized worse than those found by SGD.
  • For smooth convex functions with exact gradients, Nesterov's method reduces the error at rate O(1/k²) after k steps, against O(1/k) for gradient descent, which is the best rate any first-order method can guarantee on this class (Nesterov, 1983).

Questions to Think About

  • In a long, narrow valley of the loss, why does momentum speed up progress along the valley floor while damping the zig-zag across it?
  • Why does evaluating the gradient at the look-ahead point let Nesterov momentum correct a poor velocity sooner, making it more stable than classical momentum at high μ (Sutskever et al., 2013)?

Interactive Demo

Core Math

If you prefer intuition first, try the demo above, then come back for the math.

Key Equation
vt+1=μvt+∇L,θt+1=θt−ηvt+1v_{t+1} = \mu v_t + \nabla L, \quad \theta_{t+1} = \theta_t - \eta v_{t+1}

Stochastic gradient descent (SGD) steps against the gradient gtg_t of the loss on the mini-batch BtB_t:

θt+1=θt−η gt,gt=∇LBt(θt)\theta_{t+1} = \theta_t - \eta\, g_t, \quad g_t = \nabla L_{B_t}(\theta_t)

Momentum (the heavy-ball method; Polyak, 1964) keeps a velocity vv, an exponentially weighted sum of past gradients:

vt+1=μvt+gt,θt+1=θt−η vt+1v_{t+1} = \mu v_t + g_t, \qquad \theta_{t+1} = \theta_t - \eta\, v_{t+1}

Nesterov momentum evaluates the gradient at the point the velocity is about to carry the parameters to (Nesterov, 1983; Sutskever et al., 2013):

vt+1=μvt+∇LBt(θt−ημvt),θt+1=θt−η vt+1v_{t+1} = \mu v_t + \nabla L_{B_t}(\theta_t - \eta \mu v_t), \qquad \theta_{t+1} = \theta_t - \eta\, v_{t+1}

If the gradient stays roughly constant at gg, the velocity approaches g/(1−μ)g/(1-\mu), so each step approaches ηg/(1−μ)\eta g/(1-\mu). With μ=0.9\mu = 0.9 that is ten times the plain SGD step. Along directions where the gradient keeps changing sign, the contributions largely cancel instead.

Key Papers and Books

On the importance of initialization and momentum in deep learning

Sutskever et al.2013ICML
Read paper →

A method for unconstrained convex minimization problem with the rate of convergence O(1/k²)

Nesterov1983Soviet Mathematics Doklady
Read paper →

Some methods of speeding up the convergence of iteration methods

Polyak1964USSR Computational Mathematics and Mathematical Physics
Read paper →

Deep Learning, Sections 8.3.2–8.3.3: Momentum and Nesterov Momentum

Goodfellow, Bengio & Courville2016MIT Press (textbook)
View book →

Connections

Prerequisites

BackpropagationSharpness