Continuous Function

Foundation reference

Weight Decay & AdamW: Decoupled Regularization

Weight decay shrinks the weights at each update; AdamW applies this shrinkage separately from Adam’s per-parameter scaling. The published GPT-3 and LLaMA recipes both used Adam with decoupled weight decay of 0.1 (Brown et al., 2020; Touvron et al., 2023).

A newer notebook covers this concept. This older page keeps its summary, math and sources.

Concept 58 of 100OptimizationStudy phase 3
Open the notebook demoOpen the newer notebookAll concepts
Key equation
θt+1=(1−ηλ) θt−η m^tv^t+ϵ\theta_{t+1} = (1 - \eta\lambda)\,\theta_t - \eta\, \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}
Reading map and next steps

Study guide

Where to start

Concept 58 of 100Weight Decay & AdamWOptimization · Study phase 3: Optimization & generalization
Question

Why does Adam's per-coordinate scaling also rescale an L2 penalty that is added to the gradient, so that weights with a history of large gradients are pulled toward zero less?

Open the notebook demoGo to the mathOpen the newer notebookNext concept
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.

Before you use the demo in the newer notebook, say what you expect to change.

CheckCheck against the math and the papers.

Compare what the notebook demo shows with the key equation here and with the papers cited below.

GeneralizeName what stays the same.

Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.

ContinueContinue in the newer notebook.

The newer notebook covers the same concept and has an interactive demo.

Why It Matters for Modern Models

  • Weight decay shrinks the weights at each update; AdamW applies this shrinkage separately from Adam’s per-parameter scaling. The published GPT-3 and LLaMA recipes both used Adam with decoupled weight decay of 0.1 (Brown et al., 2020; Touvron et al., 2023).
  • Common implementations of Adam applied "weight decay" as an L2 term added to the gradient. Loshchilov & Hutter (2019) showed that this is not equivalent, and that decoupling the decay improved Adam's generalization enough to compete with SGD with momentum on image classification.
  • Decoupling also makes the best weight-decay value largely independent of the learning rate, which simplifies tuning (Loshchilov & Hutter, 2019).

Questions to Think About

  • What changes when the decay term λθ is applied to the weights directly instead of passing through Adam's division by √v̂?
  • When a layer is followed by normalization, scaling its weights does not change the output; why does weight decay still matter there, through its effect on the effective learning rate (Zhang et al., 2019)?

Interactive Demo

The interactive demo for this concept is in the newer notebook.

Open the notebook demo

Core Math

If you prefer intuition first, try the demo in the newer notebook, then come back for the math.

Key Equation
θt+1=(1−ηλ) θt−η m^tv^t+ϵ\theta_{t+1} = (1 - \eta\lambda)\,\theta_t - \eta\, \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}

L2 regularization adds a penalty on the size of the weights to the loss:

Lreg(θ)=L(θ)+λ2∥θ∥2,∇Lreg=∇L+λθL_{\text{reg}}(\theta) = L(\theta) + \frac{\lambda}{2} \|\theta\|^2, \qquad \nabla L_{\text{reg}} = \nabla L + \lambda \theta

Weight decay shrinks the weights directly at every step:

θt+1=(1−ηλ) θt−η ∇L(θt)\theta_{t+1} = (1 - \eta \lambda)\, \theta_t - \eta\, \nabla L(\theta_t)

For plain SGD the two give the same update. For Adam they do not: an L2 term added to the gradient is divided by v^t\sqrt{\hat v_t} along with the rest of the gradient, so weights with large past gradients are regularized less (Loshchilov & Hutter, 2019).

AdamW decouples the decay from the adaptive step. With mtm_t and vtv_t the moving averages of ∇L\nabla L and (∇L)2(\nabla L)^2, and m^t\hat m_t, v^t\hat v_t their bias-corrected versions,

θt+1=θt−η(m^tv^t+ϵ+λθt)\theta_{t+1} = \theta_t - \eta \left( \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} + \lambda \theta_t \right)

so every weight decays at the same rate ηλ\eta\lambda per step, whatever its gradient history.

Key Papers and Books

Decoupled Weight Decay Regularization

Loshchilov & Hutter2019ICLR
Read paper →

Three Mechanisms of Weight Decay Regularization

Zhang et al.2019ICLR
Read paper →

Deep Learning, Section 7.1.1: L2 Parameter Regularization

Goodfellow, Bengio & Courville2016MIT Press (textbook)
View book →

Connections

Prerequisites

AdamMaximum Likelihood

Where to go next

A newer notebook covers Weight Decay & AdamW, with an interactive demo.

Continue in the newer notebook