Weight Decay & AdamW: Decoupled Regularization

Let $L(\theta)$ be the task loss and $\lambda > 0$ the regularization strength.

With an $L_2$ penalty, the regularized objective is

so the gradient becomes

For SGD, this leads to

which is exactly a weight-decay step.

For Adam, the adaptive preconditioner changes things. AdamW writes the update as

The shrinkage term $-\eta \lambda \theta_t$ is not divided by $\sqrt{\hat v_t} + \epsilon$, so every coordinate decays at the same relative rate $\eta\lambda$ instead of a rate set by its gradient history. (Here $\hat m_t$ and $\hat v_t$ are the bias-corrected averages defined on the Adam page.)

Conventions for $\lambda$ differ, so compare values only within one. Loshchilov and Hutter (2019) write weight decay per step, $\theta_{t+1}=(1-\lambda)\theta_t-\alpha\nabla f_t(\theta_t)$, which for plain SGD equals an $L_2$ penalty with coefficient $\lambda/\alpha$ (their Proposition 1); in their AdamW (Algorithm 2) the decay is scaled by a schedule multiplier, not by the learning rate. PyTorch's AdamW multiplies $\lambda$ by the learning rate, as written above.

One step with fixed per-coordinate factors

To see the difference without the moving averages, freeze Adam's per-coordinate factors for one step as a diagonal matrix $P$; in Adam, $P$ would hold $1/(\sqrt{\hat v_t}+\epsilon)$. Each method's step is a task part plus a regularization part, both computed from the same pre-update $\theta$:

Take $\theta=(2,2)$, $g=(1,-1)$, $\eta=1/10$, $\lambda=1/5$ and $P=\operatorname{diag}(4,1/4)$. The task part is $(-2/5,\,1/40)$. The $L_2$ part is $(-4/25,\,-1/100)$ and the decoupled part is $(-1/25,\,-1/25)$: the penalty pulls four times harder where $P=4$ and four times more weakly where $P=1/4$. The new weights are $(36/25,\,403/200)=(1.44,\,2.015)$ with the $L_2$ penalty and $(39/25,\,397/200)=(1.56,\,1.985)$ with decoupled decay. In the second coordinate the $L_2$ part points towards zero, yet the weight grows from $2$ to $2.015$, because the task part is larger: a pull towards zero is not the whole step.

Changing only $P_{11}$ from $4$ to $1$ makes the first coordinate agree ($93/50$ under both methods) while the second still differs. By the difference formula, with the same $\lambda$ the two methods take the same step for every $\theta$ and $g$ only when $P=I$. A uniform $P=sI$ can be matched by using $\lambda/s$ in the penalty, but no single coefficient matches a $P$ whose diagonal entries differ; this is Loshchilov and Hutter's Proposition 2, and their Proposition 3 interprets decoupled decay under a fixed $P$ as a rescaled $L_2$ penalty. In Adam, $P$ changes at every step, and an $L_2$ term added to the gradient also enters both moving averages.

The first Adam step

From zero averages and with $\epsilon=0$, Adam's first step is $-\eta\,u/|u|$ in each coordinate, where $u$ is the gradient it receives (see the Adam page). With the $L_2$ penalty $u=g+\lambda\theta$; in the example $u=(7/5,\,-3/5)$, with the same signs as $g$, so the step is $(-1/10,\,1/10)$, exactly as with no penalty at all. AdamW takes the same Adam step and subtracts $\eta\lambda\theta=(1/25,\,1/25)$, giving $(-7/50,\,3/50)$. At this step the penalty disappears into Adam's normalization unless it reverses a gradient's sign; later it does not vanish, but it is still divided by $\sqrt{\hat v_t}$.