Optimizers Overview

An optimizer is a rule for moving a model's parameters from $w_t$ to $w_{t+1}$ so as to reduce the loss $L$ . The optimizers below (gradient descent, momentum, and adaptive updates such as RMSProp) differ mainly in two choices: the direction of each step and its size. The figure runs gradient descent with and without momentum on a one-dimensional quadratic, and changing the learning rate and momentum shows how each affects the path to the minimum. You will need derivatives and the idea of a gradient.

The Basics: Gradient Descent

Gradient descent computes the gradient of the loss with respect to the parameters and takes a step in the opposite direction, the direction in which the loss falls fastest. The learning rate $\eta_t$ sets the size of the step:

w_{t+1} = w_t - \eta_t \nabla_w L(w_t)

Here $L$ is the loss averaged over the whole training set, so every step needs a pass over all of the data.

Stochastic Gradient Descent (SGD)

In practice, computing the gradient over the entire dataset is expensive. SGD estimates it from a random mini-batch of examples:

w_{t+1} = w_t - \eta \nabla_w L_{B_t}(w_t)

where $B_t$ is the mini-batch drawn at step $t$ and $L_{B_t}$ is the loss averaged over it. Because the batch is drawn at random, the mini-batch gradient is a noisy but unbiased estimate of the full gradient, and each step costs only a small fraction of a pass over the data.

Momentum

Plain SGD can zig-zag in long, narrow valleys, where the gradient points mostly across the valley rather than along it. SGD with momentum adds a velocity term that accumulates past gradients (Polyak, 1964):

\begin{aligned} v_{t+1} &= \beta v_t + \nabla_w L_{B_t}(w_t) \\ w_{t+1} &= w_t - \eta v_{t+1} \end{aligned}

A typical choice is $\beta \approx 0.9$ . The velocity is an exponentially weighted sum of past gradients, which is why the method is often pictured as a heavy ball rolling downhill:

Components that point the same way from step to step add up, by as much as a factor $1/(1-\beta) = 10$ for $\beta = 0.9$
Components that flip sign from step to step, such as the zig-zag across a valley, partly cancel, settling at about $1/(1+\beta) \approx 0.53$ times a single gradient
Gradient noise from the mini-batches is averaged over roughly the last $1/(1-\beta)$ steps

Nesterov Momentum (NAG)

Nesterov accelerated gradient (NAG) changes where the gradient is evaluated:

Look ahead to where the current velocity would carry the parameters
Measure the gradient at that look-ahead point
Update the velocity with that gradient

\begin{aligned} v_{t+1} &= \beta v_t + \nabla_w L(w_t - \eta \beta v_t) \\ w_{t+1} &= w_t - \eta v_{t+1} \end{aligned}

On smooth convex problems, Nesterov's method, with a suitable momentum schedule, provably converges faster than plain gradient descent: the gap to the lowest loss shrinks like $1/t^2$ instead of $1/t$ (Nesterov, 1983). Sutskever et al. (2013) showed that this form of momentum also works well for training deep networks.

RMSProp

RMSProp (Root Mean Square Propagation), proposed by Hinton in his lecture notes (Tieleman & Hinton, 2012), keeps an exponential moving average of squared gradients and divides each update by its square root:

v_t = \beta v_{t-1} + (1-\beta) g_t^2, \quad w_{t+1} = w_t - \eta \frac{g_t}{\sqrt{v_t} + \epsilon}

where $g_t = \nabla_w L(w_t)$ is the gradient, $\beta \approx 0.9$ is the decay rate, and $\epsilon \approx 10^{-8}$ prevents division by zero. Squares, roots and division act on each parameter separately.

Intuition: divide by a running estimate of how large this parameter's gradients usually are. Parameters with consistently large gradients get smaller effective learning rates; parameters with small gradients get larger ones.

Adam keeps this per-parameter scaling and adds momentum.

Families of Optimizers

We can group common optimizers into a few families:

Plain SGD: uses the raw mini-batch gradient, with a learning rate that may follow a schedule
Momentum / Nesterov: smooth gradients over time and follow the accumulated direction
Adaptive methods: keep per-parameter statistics to scale each parameter's step (AdaGrad, Duchi et al., 2011; RMSProp; Adam and AdamW)
Matrix-aware methods: use the fact that a weight matrix is a linear map, not just a list of numbers. Shampoo (Gupta et al., 2018) preconditions each side of the matrix; the Muon optimizer orthogonalizes the update matrix

Try It Yourself

Gradient Descent Playground

Adjust the learning rate η and momentum μ to see how the optimizer moves along a simple 1D loss, L(x) = ½(x − 2)², whose minimum is at x = 2.

Try:

Learning rate (0.20)Momentum (0.80)

Step: 0

x: 4.000

Loss: 2.0000

On this bowl, plain gradient descent overshoots the minimum only when η > 1 and diverges only when η > 2, beyond this slider's range. Momentum carries velocity from step to step: a little speeds up the early steps, a lot makes the path swing past the minimum before it settles.

The playground minimizes the one-dimensional quadratic $L(x) = \tfrac{1}{2}(x - 2)^2$ , whose curvature is 1. Without momentum, each step multiplies the distance to the minimum by $1 - \eta$ , so gradient descent converges for any $\eta$ between 0 and 2 and does not oscillate while $\eta < 1$ . With momentum it converges whenever $\eta < 2(1 + \beta)$ . Every setting the sliders allow ( $\eta \le 1$ , $\beta < 1$ ) therefore converges; a large $\beta$ makes the point overshoot and swing back and forth on the way.

When to Use SGD + Momentum

SGD with momentum remains common for convolutional image classifiers, usually with a step or cosine learning-rate schedule. Wilson et al. (2017) found that on several image-classification and language tasks, well-tuned SGD reached better test accuracy than adaptive methods.

For transformers, Adam-type methods are the usual choice. Zhang et al. (2020) give one reason: gradient noise in attention models is heavy-tailed, which hurts plain SGD more than adaptive methods or SGD with gradient clipping.

AdamW builds on these ideas with per-parameter scaling and decoupled weight decay. Muon keeps momentum but orthogonalizes the update of each weight matrix.

Choosing an Optimizer

There's no universally best optimizer. In a benchmark of fifteen popular optimizers, Schmidt et al. (2021) found that performance varied greatly across tasks and that no optimizer clearly dominated; trying several optimizers at their default settings worked about as well as tuning one of them, and Adam remained a strong choice. Newer methods target particular settings, such as LAMB (You et al., 2020) for very large batches and Muon for the weight matrices of hidden layers.

Sources

Polyak, B. T. (1964). Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics 4(5). doi.org/10.1016/0041-5553(64)90137-5
Nesterov, Y. E. (1983). A method of solving a convex programming problem with convergence rate O(1/k²). Doklady Akademii Nauk SSSR 269(3), 543–547. mathnet.ru/eng/dan46009
Sutskever, I., et al. (2013). On the importance of initialization and momentum in deep learning. ICML. proceedings.mlr.press/v28/sutskever13.html
Tieleman, T., & Hinton, G. (2012). Lecture 6.5, RMSProp: Divide the gradient by a running average of its recent magnitude. Coursera: Neural Networks for Machine Learning.
Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. JMLR 12. jmlr.org/papers/v12/duchi11a.html
Gupta, V., Koren, T., & Singer, Y. (2018). Shampoo: Preconditioned Stochastic Tensor Optimization. ICML. arxiv.org/abs/1802.09568
Wilson, A. C., et al. (2017). The Marginal Value of Adaptive Gradient Methods in Machine Learning. NeurIPS. arxiv.org/abs/1705.08292
Zhang, J., et al. (2020). Why are Adaptive Methods Good for Attention Models? NeurIPS. arxiv.org/abs/1912.03194
Schmidt, R. M., Schneider, F., & Hennig, P. (2021). Descending through a Crowded Valley: Benchmarking Deep Learning Optimizers. ICML. arxiv.org/abs/2007.01547
You, Y., et al. (2020). Large Batch Optimization for Deep Learning: Training BERT in 76 minutes. ICLR. arxiv.org/abs/1904.00962
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning, chapter 8: Optimization for Training Deep Models. MIT Press. deeplearningbook.org