Gradient Clipping & Explosion Prevention

Let $g \in \mathbb{R}^d$ be the gradient and let $c > 0$ be the clipping threshold.

Global norm clipping defines

Equivalently,

If your optimizer step is $\Delta \theta = -\eta \tilde g$, then whenever clipping activates,

So clipping puts a hard cap on the update size.

Why can gradients explode in the first place? In a deep network, gradients are products of Jacobians:

If those Jacobians repeatedly amplify norms, backpropagated signals can grow exponentially with depth or sequence length.

Which vector is bounded

Clipping acts on the gradient before the optimizer uses it. The simplest thing an optimizer can then do is rescale each coordinate of the clipped gradient by a fixed factor. Write those factors as a matrix $P$, so that

where $\lVert P\rVert_2$ is the largest factor by which $P$ can stretch a vector. Plain gradient descent is $P=I$, and the bound becomes the equality above whenever clipping is active.

Take $g=(6,8)$, $c=5$ and $\eta=1/10$. Then $\lVert g\rVert_2=10$, the scale is $1/2$ and $\tilde g=(3,4)$, of length exactly $5$. Plain gradient descent steps by $(-3/10,-2/5)$, of length $1/2=\eta c$, where without clipping it would have stepped by $(-3/5,-4/5)$, of length $1$. Now let $P=\operatorname{diag}(2,1/2)$. Then $q=(6,2)$ and $\Delta\theta=(-3/5,-1/5)$, of length $\sqrt{10}/5\approx0.632$. The clip is still correct, but the step is longer than $\eta c$ and no longer points along $-g$; it stays within the bound $\eta\lVert P\rVert_2c=1$. A uniform scaling $P=sI$ would keep the direction and multiply the length by $s$. When the gradient's norm equals $c$ exactly, the scale is $1$ whether an implementation rescales at $\lVert g\rVert_2>c$ or at $\lVert g\rVert_2\ge c$.

Adam does more than rescale. Its step divides a running average of past gradients by the root of a running average of their squares, so it depends on the clipped gradients of earlier steps too: with a zero gradient at one step, Adam still moves whenever its average of earlier gradients is not zero, while the bound above would allow no movement. Clipping therefore bounds what enters Adam's averages, not Adam's step.

In PyTorch, torch.nn.utils.clip_grad_norm_ computes one norm over the gradients of all the parameters given to it, as if they were concatenated into a single vector, and rescales them in place; the optimizer step then runs on the clipped gradients.