Learning Rate Schedules: Warmup, Decay & Cycling

Why It Matters for Modern Models

A learning-rate schedule sets the step size at each training update. With the same model and data, the choice of schedule can determine whether training reduces the loss or diverges.
Warmup avoids early instability: in the first steps Adam's second-moment estimate rests on very few gradients, so its effective step sizes vary widely, and a small initial rate keeps those steps safe (Liu et al., 2020).
Linear warmup followed by cosine decay is a common choice for language-model pretraining, used for example by GPT-3 and LLaMA (Brown et al., 2020; Touvron et al., 2023).

Questions to Think About

How do cosine and linear decay differ in how long they keep the rate near its peak and how long they spend near its minimum?
How does the learning-rate range test find a good peak rate by training briefly while increasing the rate until the loss starts to rise (Smith, 2017)?

Interactive Demo

The interactive demo for this concept is in the newer notebook.

Open the notebook demo

Core Math

If you prefer intuition first, try the demo in the newer notebook, then come back for the math.

Key Equation

\eta_t = \eta_{\min} + \frac{1}{2}(\eta_{\max} - \eta_{\min})\left(1 + \cos\left(\frac{\pi t}{T}\right)\right)

A learning-rate schedule sets the step size $\eta_t$ at each training step $t$ .

Linear warmup raises the rate from 0 to its peak over the first $T_w$ steps (Goyal et al., 2017):

\eta_t = \eta_{\max} \cdot \frac{t}{T_w}, \quad t \leq T_w

Cosine decay then lowers it smoothly to $\eta_{\min}$ by the final step $T$ (Loshchilov & Hutter, 2017):

\eta_t = \eta_{\min} + \frac{1}{2}(\eta_{\max} - \eta_{\min})\left(1 + \cos\left(\frac{t - T_w}{T - T_w}\pi\right)\right), \quad T_w < t \le T

With $T_w = 0$ this is the key equation above. SGDR restarts the cosine cycle periodically ("warm restarts"); cyclical schedules move the rate up and down between bounds (Smith, 2017).

Inverse-square-root decay after warmup was the schedule of the original Transformer (Vaswani et al., 2017):

\eta_t \propto \min\!\left(t^{-1/2},\ t\, T_w^{-3/2}\right)

Many published language-model recipes use linear warmup followed by cosine decay to about 10% of the peak rate, for example GPT-3 and LLaMA (Brown et al., 2020; Touvron et al., 2023). Warmup–stable–decay schedules instead hold the rate constant for most of training and decay it only at the end (Hu et al., 2024).