Gradient clipping caps a gradient’s norm or its individual entries before an update. It limits the effect of occasional large gradients; GPT-3 and LLaMA both clipped the global gradient norm at 1.0 (Brown et al., 2020; Touvron et al., 2023).
A newer notebook covers this concept. This older page keeps its summary, math and sources.
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you use the demo in the newer notebook, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the notebook demo shows with the key equation here and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueContinue in the newer notebook.
The newer notebook covers the same concept and has an interactive demo.
Why It Matters for Modern Models
Gradient clipping caps a gradient’s norm or its individual entries before an update. It limits the effect of occasional large gradients; GPT-3 and LLaMA both clipped the global gradient norm at 1.0 (Brown et al., 2020; Touvron et al., 2023).
It does not remove every instability: PaLM's largest model showed about 20 spikes in its training loss even with gradient clipping enabled (Chowdhery et al., 2023).
The same product of Jacobians that explains exploding gradients also explains vanishing ones, which clipping cannot fix; that problem is addressed by gated recurrent units, residual connections and careful initialization.
Questions to Think About
When one batch produces a gradient many times larger than usual, how does clipping bound how far a plain SGD update can move the weights?
How can tracking the gradient norm during training help detect instabilities, such as loss spikes, before they derail a run?
Interactive Demo
The interactive demo for this concept is in the newer notebook.
If you prefer intuition first, try the demo in the newer notebook, then come back for the math.
Key Equation
g~=min(1,c/∥g∥)⋅g
Gradient norm clipping rescales the whole gradient g when its norm exceeds a threshold c:
g~={gc⋅∥g∥gif ∥g∥≤cif ∥g∥>c
The direction is unchanged; only the length is capped. In language-model training the norm is usually taken over all parameters together (the global norm).
Value clipping clips each coordinate separately, g~i=clip(gi,−c,c), which can change the direction of g.
Why gradients explode. Backpropagation through D layers hl=fl(hl−1) multiplies their Jacobians:
∂h0∂L=∂hD∂L∂hD−1∂hD⋯∂h0∂h1
If these Jacobians stretch vectors by a factor greater than 1 again and again, the product can grow exponentially with depth, or with the number of time steps in a recurrent network. For a recurrent network, Pascanu et al. (2013) show that explosion requires the largest singular value of the recurrent weight matrix to exceed 1/γ, where γ bounds the derivative of the activation function.
Key Papers and Books
On the difficulty of training Recurrent Neural Networks