Bring the mental model from Adam Optimizer; this page will reuse it instead of restarting from zero.
Learning Rate Schedules: Warmup, Decay & Cycling
Schedule shapes that change the scalar learning-rate scale over training, with sourced CLR/range-test and SGDR cosine-restart examples plus caveated warmup/decay teaching patterns.
Intuition
Build the mental picture first so the rest of the page has something to attach to.
The learning rate is a scalar multiplier in an optimizer update. In plain gradient descent it directly scales the gradient step; adaptive optimizers also transform the gradient before applying that multiplier. Changing the multiplier over a finite run changes how large the optimizer's applied updates can be at different times.
Two page-local teaching patterns, treated here as extensions beyond the listed Smith/SGDR sources:
- Warmup: start small, then ramp up. This page uses warmup as a teaching pattern for smaller early update scale, but the listed Smith/SGDR sources do not by themselves source large-model warmup practice or Adam warmup rationale.
- Decay: reduce the learning-rate multiplier later so the scalar update scale is smaller late in training.
This page includes warmup followed by decay as a teaching shape, but this source-checked claim focuses only on learning-rate range tests, cyclical policies, and cosine annealing/restarts.
Copy-only prompts — each action copies a page-grounded prompt to your clipboard. Nothing is sent by this site.
Math
Translate the story into symbols, assumptions, and a derivation you can inspect.
Indexing note. This page defines exactly learning rates applied to optimizer updates . Assume integers and , with . This finite, endpoint-inclusive definition is a page-local teaching convention, not a universal scheduler API convention; libraries differ over whether a boundary value is an applied update or a scheduler state before or after one. The warmup omits the conceptual zero endpoint, so its first applied rate is positive when ; admits the degenerate all-zero schedule.
For a decay coordinate , define
Choose or . When , there is no warmup phase, and spans all applied updates:
When , the first applied updates form a linear warmup that omits the conceptual zero endpoint, followed by decay:
Update reaches and is the shared endpoint of warmup and decay. When , update uses and is already below the peak, so the peak is not repeated. When , the post-warmup segment instead remains constant at . The final applied update, , uses and reaches . Here is the decay endpoint, not necessarily the minimum over the warmup-and-decay sequence: an early positive warmup rate can be smaller.
A separate page-local teaching alternative is inverse-square-root decay, where is the rate at update :
For every displayed family, the demo's reported mean is the discrete mean over all applied updates,
not the mean of the smaller set of points used to draw the chart. SGDR supplies the within-cycle half-cosine shape and its maximum/minimum endpoints; it does not supply this page's linear warmup or finite-update indexing.
Copy-only prompts — each action copies a page-grounded prompt to your clipboard. Nothing is sent by this site.
Code
Keep the implementation aligned with the notation so the algorithm is legible.
import math
def warmup_cosine(T, Tw, eta_max, eta_min):
if not isinstance(T, int) or isinstance(T, bool) or T < 2:
raise ValueError("T must be an integer with T >= 2")
if not isinstance(Tw, int) or isinstance(Tw, bool) or not 0 <= Tw < T:
raise ValueError("Tw must be an integer with 0 <= Tw < T")
if not all(math.isfinite(v) for v in (eta_max, eta_min)):
raise ValueError("learning-rate bounds must be finite")
if not 0 <= eta_min <= eta_max:
raise ValueError("require 0 <= eta_min <= eta_max")
lr = [0.0] * T
for t in range(T):
if Tw > 0 and t < Tw:
lr[t] = eta_max * (t + 1) / Tw
else:
u = t / (T - 1) if Tw == 0 else (t - Tw + 1) / (T - Tw)
lr[t] = eta_min + 0.5 * (eta_max - eta_min) * (1 + math.cos(math.pi * u))
return lr
T, Tw = 10_000, 500
lr = warmup_cosine(T=T, Tw=Tw, eta_max=3e-4, eta_min=3e-5)
print("lr[0], lr[Tw-1], lr[Tw], lr[-1]:", lr[0], lr[Tw - 1], lr[Tw], lr[-1])
print("avg lr over all applied updates:", sum(lr) / len(lr))
Copy-only prompts — each action copies a page-grounded prompt to your clipboard. Nothing is sent by this site.
Interactive Demo
Use direct manipulation to connect the explanation to a moving system.
Live Concept Demo
Explore Learning Rate Schedules: Warmup, Decay & Cycling
The stage is code-native and interactive. Use it to test the explanation against the mechanism.
Manipulate one control and predict the visible change.
Choose what to inspect in Learning Rate Schedules: Warmup, Decay & Cycling. This shared fallback is an observation guide, not evidence of learning.
Predict the schedule's behavior, then reveal the finite applied-update sequence. Tune warmup length and decay style while watching the exact first rate, mean over all updates, final rate, and the boundary between the last warmup update and the first decay update. The chart uses a smaller anchored sample only for drawing.
Copy-only prompts — each action copies a page-grounded prompt to your clipboard. Nothing is sent by this site.
Concept: Learning Rate Schedules: Warmup, Decay & Cycling
What is the smallest example that makes Learning Rate Schedules: Warmup, Decay & Cycling click without losing the math?
Object contextOptimization
concept:optimization/learning-rate-schedulesLearning Rate Schedules: Warmup, Decay & Cycling
What is the smallest example that makes Learning Rate Schedules: Warmup, Decay & Cycling click without losing the math?
Start with the prediction checkpoint, then compare the reveal to the mental model.
Take this moveStudy modes
Keep the object fixed; change the lens.Route back through the notebook
Carry the same object through intuition, math, code, and demo.
Schedule shapes that change the scalar learning-rate scale over training, with sourced CLR/range-test and SGDR cosine-restart examples plus caveated warmup/decay teaching patterns.
The next edge should feel earned: use the demo prediction here before following Scaling Laws & Emergent Abilities.
After The First Pass
Turn the concept into an inspected object.
The lower panels are one second act: keep the object fixed, inspect it visually, check source boundaries, practice transfer, then attach the research question.Mechanism Storyboard
See the idea move before the page explains it
Schedule shapes that change the scalar learning-rate scale over training, with sourced CLR/range-test and SGDR cosine-restart examples plus caveated warmup/decay teaching patterns.

Start with the picture, metaphor, or geometric mechanism.
Before reading further, choose the kind of change Learning Rate Schedules: Warmup, Decay & Cycling should make visible.
Visual Inquiry
Make the image answer a mathematical question
Schedule shapes that change the scalar learning-rate scale over training, with sourced CLR/range-test and SGDR cosine-restart examples plus caveated warmup/decay teaching patterns.
Which visible object should carry the first intuition?
Pick the cue that should make Learning Rate Schedules: Warmup, Decay & Cycling easier to reason about before the page gives the answer.
Source Grounding
Canonical references for the mechanism on this page.
What is the smallest example that makes Learning Rate Schedules: Warmup, Decay & Cycling click without losing the math?
concept:optimization/learning-rate-schedulessources: smith-2015-cyclical-learning-rates, loshchilov-2016-sgdr
Open the closest source note before trusting the local explanation.
2 selected-object sources shown first; 2 references total.
Audit the claim boundary, then ask from the same selected object.
Grounds LR range tests and cyclical policies that vary the global learning rate between lower and upper bounds.
Smith supports LR range tests and CLR between boundary values. SGDR supports cosine annealing between maximum and minimum endpoints within a restart cycle. The finite-update indexing, sha...
Does not check warmup practice or rationale, inverse-sqrt schedules, a universal scheduler-library indexing convention, convergence guarantees, or schedule superiority. Source support cov...
Grounds the endpoint-inclusive half-cosine shape within SGD warm-restart cycles. It does not define this page's linear warmup or its enumeration of exactly T applied updates.
Smith supports LR range tests and CLR between boundary values. SGDR supports cosine annealing between maximum and minimum endpoints within a restart cycle. The finite-update indexing, sha...
Does not check warmup practice or rationale, inverse-sqrt schedules, a universal scheduler-library indexing convention, convergence guarantees, or schedule superiority. Source support cov...
Claim Review
Schedule shapes that change the scalar learning-rate scale over training, with sourced CLR/range-test and SGDR cosine-restart examples plus caveated warmup/decay teaching patterns.
What is the smallest example that makes Learning Rate Schedules: Warmup, Decay & Cycling click without losing the math?
concept:optimization/learning-rate-schedulessources: smith-2015-cyclical-learning-rates, loshchilov-2016-sgdr
Treat every claim as provisional until source support and a local witness agree.
1 structured claim check on this concept.
Run the prediction or practice transfer before asking for a grounded review.
Publisher-side editorial review is not independent replication. Claims without it still need exact source-support review. 2 references and 3 local witnesses are available for inspection.
Smith supports LR range tests and CLR between boundary values. SGDR supports cosine annealing between maximum and minimum endpoints within a restart cycle. The finite-update indexing, shared boundary, and ex...
Does not check warmup practice or rationale, inverse-sqrt schedules, a universal scheduler-library indexing convention, convergence guarantees, or schedule superiority. Source support covers only LR range te...
Smith supports scalar LR as a key hyperparameter, CLR between bounds, and an LR range test for selecting bounds. Loshchilov and Hutter support warm restarts and half-cosine annealing from eta_max to eta_min within a cycle. Neither paper defines this page's linear warmup, inverse-sqrt family, finite applied-update indexing, convergence behavior, or schedule superiority; those claims are excluded or labeled as page-local teaching definitions.
Reviewer: codex+gpt-pro; reviewed 2026-08-21Practice · Learning Rate Schedules: Warmup, Decay & Cycling
Try the idea in your own words
Schedule shapes that change the scalar learning-rate scale over training, with sourced CLR/range-test and SGDR cosine-restart examples plus caveated warmup/decay teaching patterns.
Concept · Current object
Learning Rate Schedules: Warmup, Decay & Cycling
Source boundary: sources: smith-2015-cyclical-learning-rates, loshchilov-2016-sgdr
Object context and links
Optimization
concept:optimization/learning-rate-schedulesExplain the mechanism
For Learning Rate Schedules: Warmup, Decay & Cycling: What is the smallest example that makes Learning Rate Schedules: Warmup, Decay & Cycling click without losing the math? Explain your answer, including what changes, why, and which assumption matters.
A rough first thought is enough. Your draft stays when you change tasks.
Local to this page session. Not saved after leaving or reloading.
A little help · Explain
Open one hint at a time. These are suggestions, not your answer or a grade.
0 of 3 hints shown for this question.
Clearing your answer does not erase help history. Outside help cannot be verified here.
Where am I stuck? (optional)
Take your draft to a feedback conversation
No AI feedback runs here. You can copy a prompt to use elsewhere; nothing is sent automatically. Review the text before sharing, and leave out private information.
Write an attempt before copying a feedback prompt.
This draft and any AI response do not establish mastery. Try a different case later without help; this page has not measured that learning.
- ObjectConceptLearning Rate Schedules: Warmup, Decay & Cycling
- PredictBefore revealLearning Rate Schedules: Warmup, Decay & Cycling prediction
- WitnessCompare codeLearning Rate Schedules: Warmup, Decay & Cycling code witness 1
- RoomAsk groundedChecking local snapshot
Research Room
Attach the question to an exact object
Pick the concept, equation, source, code witness, claim, misconception, or demo state before asking for help. The handoff stays grounded to that object.Open the draft below to save one note and next action in this browser.
Learning Rate Schedules: Warmup, Decay & Cycling
What is the smallest example that makes Learning Rate Schedules: Warmup, Decay & Cycling click without losing the math?
These are fixed, deterministic perspectives derived from the selected object. They do not represent people, community contributions, or independent review.
Source ids smith-2015-cyclical-learning-rates, loshchilov-2016-sgdr must support the exact object, not just the surrounding topic.
Treat this as a mechanism object: connect the definition to one equation, code witness, or demo before broadening the discussion.
Ask the learner to perturb one representation, then check whether the same invariant survives in math, code, and demo.
The learner can state the mechanism in their own words
Local action draftNo local draft saved yetExpand only when ready to capture one local next action
This draft stays locally in this browser for concept:optimization/learning-rate-schedules.
- Source ids to inspect: smith-2015-cyclical-learning-rates, loshchilov-2016-sgdr
- Definition, prerequisite, and contrast concept links
- The equation or code witness that makes the concept operational
- One demo state that shows the invariant instead of a slogan
- The learner can state the mechanism in their own words
- The learner can name the prerequisite that would repair confusion
- The learner can predict how the mechanism changes under one perturbation
I am working in Continuous Function's research reading room. Object: concept - Learning Rate Schedules: Warmup, Decay & Cycling Object key: concept:optimization/learning-rate-schedules Context: Optimization Anchor id: concept/concept-notebook/optimization/learning-rate-schedules Open question: What is the smallest example that makes Learning Rate Schedules: Warmup, Decay & Cycling click without losing the math? Evidence to inspect: - Source ids to inspect: smith-2015-cyclical-learning-rates, loshchilov-2016-sgdr - Definition, prerequisite, and contrast concept links - The equation or code witness that makes the concept operational - One demo state that shows the invariant instead of a slogan Deterministic role lenses for this object: - Boundary: fixed perspectives, not people, community contributions, or independent review - Source-checking summary: Treat this as a mechanism object: connect the definition to one equation, code witness, or demo before broadening the discussion. - Proposed experiment: Ask the learner to perturb one representation, then check whether the same invariant survives in math, code, and demo. - Teach/transfer move: Turn the mechanism into one sentence that predicts a neighboring concept. - Assumptions: - Source ids smith-2015-cyclical-learning-rates, loshchilov-2016-sgdr must support the exact object, not just the surrounding topic. - The stable content-object key lets local drafts, prompts, and route memory attach without changing the source page. - The concept explanation is local atlas prose until checked against its math, code, and source support. - Prerequisite gaps should become a repair route, not a reason to leave the object vague. - Role-lens requests: - Learner: ask for "Ask what would make "Learning Rate Schedules: Warmup, Decay & Cycling" feel predictable rather than familiar." | assumption: Source ids smith-2015-cyclical-learning-rates, loshchilov-2016-sgdr must support the exact object, not just the surrounding topic. | next action: The learner can state the mechanism in their own words - Researcher: ask for "Source ids to inspect: smith-2015-cyclical-learning-rates, loshchilov-2016-sgdr" | assumption: The stable content-object key lets local drafts, prompts, and route memory attach without changing the source page. | next action: The learner can name the prerequisite that would repair confusion - Experimenter: ask for "Choose one variable or condition to perturb before asking for an explanation." | assumption: The concept explanation is local atlas prose until checked against its math, code, and source support. | next action: The learner can predict how the mechanism changes under one perturbation - Professor: ask for "Find the smallest transferable rule a learner could reuse without the AI." | assumption: Prerequisite gaps should become a repair route, not a reason to leave the object vague. | next action: Teach or transfer: Turn the mechanism into one sentence that predicts a neighboring concept. What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action. Current deterministic role lens for this object: - Role lens: Learner - Evidence request: Ask what would make "Learning Rate Schedules: Warmup, Decay & Cycling" feel predictable rather than familiar. - Assumption to keep visible: Source ids smith-2015-cyclical-learning-rates, loshchilov-2016-sgdr must support the exact object, not just the surrounding topic. - Proposed experiment: Ask the learner to perturb one representation, then check whether the same invariant survives in math, code, and demo. - Next action: The learner can state the mechanism in their own words
concept/concept-notebook/optimization/learning-rate-schedules
concept:optimization/learning-rate-schedules