Continuous Function

Machine Learning

RNNs and Vanishing/Exploding Gradients

Unroll an RNN and watch the backward signal cross repeated Jacobians, shrinking, persisting, or exploding as each step's multiplier sits below, near, or above one.

status: reviewimportance: criticaldifficulty 4/5math: graduateread: 19mlive demo
Back to Machine LearningNext: Gradient Clipping & Explosion Prevention (review)

Concept Structure

RNNs and Vanishing/Exploding Gradients

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

2prerequisites
4next concepts
3related links

Learner Contract

What this page should let you do.

You are here becauseUnroll an RNN and watch the backward signal cross repeated Jacobians, shrinking, persisting, or exploding as each step's multiplier sits below, near, or above one.

This Machine Learning concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Test the linkManipulate one control and predict the visible change.Then continue to Gradient Clipping & Explosion Prevention (review)

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.
Claims1/1 reviewed
Sources4 cited
Codeattached
Demolive
Reviewed2026-07-02
Updatedpage 2026-07-02

Learning item flow

4/4 sections readyAsk about thisResearch room
ConceptRNNs and Vanishing/Exploding GradientsMachine LearningEquationRNNs and Vanishing/Exploding Gradients equation 1Exact equationCodeRNNs and Vanishing/Exploding Gradients runnable code 1Exact runnable codeDemoRNNs and Vanishing/Exploding Gradients interactive demoInteractive demoClaimIn an unrolled RNN, backpropagation through time sends sensitivities...Exact claim checkSourceDive into Deep Learning: Backpropagation Through TimeExact source
ConceptRNNs and Vanishing/Exploding GradientsMachine Learning
5 sources attachedLocal snapshot ready
concept:machine-learning/rnns-vanishing-exploding-gradients
CodenearbyPredictbefore revealRoomitem handoff
01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.
01

01

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

You are here because sequence models do something ordinary feedforward networks do not: they reuse the same state update again and again.

Before this, know backpropagation and activation derivatives. By the end, you should be able to unroll a recurrent neural network, point to the repeated Jacobian product, and predict when an early-time gradient becomes too tiny or too large to be useful.

An RNN carries a hidden state through time:

h0→h1→h2→⋯→hT.h_0 \rightarrow h_1 \rightarrow h_2 \rightarrow \cdots \rightarrow h_T.

That forward arrow feels like memory. The backward pass reveals the cost of that memory. If a loss at time TT needs to teach something that happened near time 00, the gradient must walk backward through each recurrent state transition.

One step is not scary. Eight steps might be fine. Hundreds of steps can become a corridor of multiplication.

  • If the local backward multipliers are usually smaller than 11, the early gradient fades.
  • If they are usually close to 11 and well conditioned, the signal can stay usable.
  • If they are usually larger than 11, the signal can explode.

This is not just an RNN history lesson. It is the sequence-model version of the same chain-rule problem you saw in activation functions and initialization. LSTM/GRU gates, gradient clipping, residual/direct paths, and attention all become easier to understand once you can see the corridor.

The caveat is important: real RNNs have vector states, nonlinearities, changing inputs, time-varying Jacobians, and losses at many positions. The demo below is a finite linearized witness for the mechanism, not a verdict on every recurrent architecture.

02

02

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt
Equation 1at=Whhht−1+Wxhxt+b,ht=ϕ(at),a_t = W_{hh}h_{t-1}+W_{xh}x_t+b, \qquad h_t = \phi(a_t),Equation 2Jt=∂ht∂ht−1=diag⁡(ϕ′(at))Whh.J_t=\frac{\partial h_t}{\partial h_{t-1}} = \operatorname{diag}(\phi'(a_t))W_{hh}.

Let the hidden state be ht∈Rdh_t\in\mathbb R^d and the input be xtx_t. A basic RNN update is

at=Whhht−1+Wxhxt+b,ht=ϕ(at),a_t = W_{hh}h_{t-1}+W_{xh}x_t+b, \qquad h_t = \phi(a_t),

where Whh∈Rd×dW_{hh}\in\mathbb R^{d\times d} is the recurrent weight matrix, WxhW_{xh} maps the input into hidden space, bb is a bias vector, and ϕ\phi is an elementwise nonlinearity such as tanh⁡\tanh.

Suppose a scalar loss LL depends on the final state hTh_T. Define the local recurrent Jacobian

Jt=∂ht∂ht−1=diag⁡(ϕ′(at))Whh.J_t=\frac{\partial h_t}{\partial h_{t-1}} = \operatorname{diag}(\phi'(a_t))W_{hh}.

Backpropagation through time applies the chain rule through the unrolled graph. For an earlier state hih_i,

∂L∂hi=(Ji+1⊤Ji+2⊤⋯JT⊤)∂L∂hT.\frac{\partial L}{\partial h_i} = \left(J_{i+1}^{\top}J_{i+2}^{\top}\cdots J_T^{\top}\right) \frac{\partial L}{\partial h_T}.

That product is the central object. Its norm can be bounded by

∥∂L∂hi∥≤(∏t=i+1T∥Jt∥)∥∂L∂hT∥.\left\|\frac{\partial L}{\partial h_i}\right\| \leq \left(\prod_{t=i+1}^{T}\|J_t\|\right) \left\|\frac{\partial L}{\partial h_T}\right\|.

This bound is not an equality in general, because matrix directions and conditioning matter. It is still the right warning sign: repeated contraction tends to erase long-range credit assignment, while repeated expansion can make updates unstable.

For the demo, we collapse the corridor into a scalar linearized toy where each local Jacobian has the same effective multiplier ρ\rho:

g0=ρTgT.g_0 = \rho^T g_T.

When ∣ρ∣<1|\rho|<1, ρT\rho^T shrinks as TT grows. When ∣ρ∣≈1|\rho|\approx 1, the signal stays near its original scale. When ∣ρ∣>1|\rho|>1, the signal grows. This is the smallest possible witness for the repeated-product mechanism.

03

03

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt
Runnable code 1import numpy as np T = 8 presets = { "vanish": 0.55, "usable": 1.0, "explode": 1.35, } def ba...python
import numpy as np

T = 8
presets = {
    "vanish": 0.55,
    "usable": 1.0,
    "explode": 1.35,
}

def backward_corridor(rho, T=8, g_T=1.0, clip=None):
    grad = g_T
    trace = [grad]
    for _ in range(T):
        grad *= rho
        if clip is not None:
            grad = np.clip(grad, -clip, clip)
        trace.append(grad)
    return np.array(trace)

for name, rho in presets.items():
    trace = backward_corridor(rho, T)
    print(name, "rho^T =", round(rho**T, 4),
          "g0 =", round(trace[-1], 4))

This code intentionally uses a scalar linearized corridor. In a real RNN, each step multiplies by a matrix Jt⊤J_t^\top, and JtJ_t changes with the hidden state, input, activation derivative, and parameters. The scalar version isolates the one thing that must not be missed: long-range gradients are products, not single slopes.

04

04

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the RNN Gradient Corridor to inspect an unrolled sequence from t=0t=0 to t=8t=8. The forward hidden states are visible, but the backward gradient trace is locked until you commit.

Before reveal, choose what happens to the target gradient ∂L/∂h0\partial L/\partial h_0:

  • Vanishes: repeated contraction pushes the gradient toward zero.
  • Stays usable: the effective multiplier stays near one.
  • Explodes: repeated expansion makes the gradient too large.

After reveal, compare the effective multiplier ∣ρ∣8|\rho|^8, the backward norm trace, and the caveat strip. This is not a verdict on every recurrent architecture; it is a finite corridor witness for repeated Jacobian products. If the path explodes, gradient clipping can bound the update norm, but clipping does not magically create long-term memory. If the path vanishes, gates and attention are motivated because they create better routes for information and gradients.

Live Concept Demo

Explore RNNs and Vanishing/Exploding Gradients

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5graduatecode-aligned
Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what RNNs and Vanishing/Exploding Gradients should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Unroll an RNN and watch the backward signal cross repeated Jacobians, shrinking, persisting, or exploding as each step's multiplier sits below, near, or above one.

Prediction open01 / Intuition
Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change RNNs and Vanishing/Exploding Gradients should make visible.

Visual Inquiry

Make the image answer a mathematical question

Unroll an RNN and watch the backward signal cross repeated Jacobians, shrinking, persisting, or exploding as each step's multiplier sits below, near, or above one.

4/4 stages readyLive demo connected
Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make RNNs and Vanishing/Exploding Gradients easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

book · 2026Dive into Deep Learning: Backpropagation Through TimeZhang, Lipton, Li, and Smola

Derives BPTT by unrolling an RNN and exposes the repeated product of recurrent derivatives/Jacobians.

Open source
book · 2026Dive into Deep Learning: Recurrent Neural NetworksZhang, Lipton, Li, and Smola

Grounds the RNN hidden-state recurrence and language-model setting.

Open source
course-notes · 2019CS224N: Language Models, RNN, GRU and LSTMStanford CS224N

Course notes for RNN language models, GRU/LSTM motivation, and the difficulty of accessing information many steps back.

Open source
paper · 2013On the difficulty of training Recurrent Neural NetworksPascanu, Mikolov, and Bengio

Primary analysis source for vanishing/exploding gradient problems in RNNs and gradient norm clipping as an exploding-gradient remedy.

Open source
paper · 1994Learning long-term dependencies with gradient descent is difficultBengio, Simard, and Frasconi

Classic source on why gradient-based learning struggles as temporal dependencies span longer intervals.

Open source

Claim Review

Unroll an RNN and watch the backward signal cross repeated Jacobians, shrinking, persisting, or exploding as each step's multiplier sits below, near, or above one.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources5 references

d2l-2026-bptt, d2l-2026-rnn, cs224n-2019-rnn-notes, pascanu-2013-training-rnns, bengio-1994-long-term-dependencies

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedIn an unrolled RNN, backpropagation through time sends sensitivities across repeated recurrent Jacobians; products with norms persistently below, near, or above one can make long-range gradients vanish, stay usable, or explode.Claim metadata: source checked

D2L explicitly derives the recurrent product in BPTT; Pascanu et al. analyze vanishing/exploding gradients in RNNs and propose gradient norm clipping for explosions; Bengio et al. ground the long-term-dependency difficulty; CS224N frames the practical RNN language-model setting and difficulty of accessing far-back information.

Sources: Dive into Deep Learning: Backpropagation Through Time, CS224N: Language Models, RNN, GRU and LSTM, On the difficulty of training Recurrent Neural Networks, Learning long-term dependencies with gradient descent is difficultFinite scalar linearized witness only; real RNN Jacobians vary with time, input, hidden state, nonlinearity, conditioning, truncation, and parameterization.A bounded review summary is present; still check caveats and exact reference scope.

Checked D2L BPTT for unrolling, chain-rule recurrence, and repeated derivative products; Pascanu/Mikolov/Bengio for vanishing/exploding RNN training difficulties and gradient norm clipping; Bengio/Simard/Frasconi for long-term dependency difficulty; and CS224N notes for the course-level RNN language-model framing. The live demo is a scalar linearized corridor, not a full nonlinear architecture benchmark. GPT Pro/Oracle publication critique remains pending.

Reviewer: codex-local-primary-source-audit; reviewed 2026-07-02
source-span-d2l-2026-bpttsource-span-cs224n-2019-rnn-notessource-span-pascanu-2013-training-rnnssource-span-bengio-1994-long-term-dependenciesEquation 1Equation 2Runnable code 1Interactive demo

Source support candidates

book 2026Dive into Deep Learning: Backpropagation Through Time

Derives BPTT by unrolling an RNN and exposes the repeated product of recurrent derivatives/Jacobians.

book 2026Dive into Deep Learning: Recurrent Neural Networks

Grounds the RNN hidden-state recurrence and language-model setting.

course-notes 2019CS224N: Language Models, RNN, GRU and LSTM

Course notes for RNN language models, GRU/LSTM motivation, and the difficulty of accessing information many steps back.

paper 2013On the difficulty of training Recurrent Neural Networks

Primary analysis source for vanishing/exploding gradient problems in RNNs and gradient norm clipping as an exploding-gradient remedy.

Mechanism checks

Equation 1
at=Whhht−1+Wxhxt+b,ht=ϕ(at),a_t = W_{hh}h_{t-1}+W_{xh}x_t+b, \qquad h_t = \phi(a_t),
Equation 2
Jt=∂ht∂ht−1=diag⁡(ϕ′(at))Whh.J_t=\frac{\partial h_t}{\partial h_{t-1}} = \operatorname{diag}(\phi'(a_t))W_{hh}.
Runnable code 1import numpy as np T = 8 presets = { "vanish": 0.55, "usable": 1.0, "explode": 1.35, } def ba...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Unroll an RNN and watch the backward signal cross repeated Jacobians, shrinking, persisting, or exploding as each step's multiplier sits below, near, or above one.

Readiness0/3 checks ready
Predict

Before touching the demo, predict one visible change that should happen in RNNs and Vanishing/Exploding Gradients.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Grounded research drawerClose
ConceptRNNs and Vanishing/Exploding GradientsMachine Learning
Runnable code comparisonRNNs and Vanishing/Exploding Gradients runnable code 1T = 8Prediction before revealRNNs and Vanishing/Exploding Gradients interactive demoManipulate one control and predict the visible change.
Grounded room questionWhat is the smallest example that makes RNNs and Vanishing/Exploding Gradients click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.
Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptMachine Learning

RNNs and Vanishing/Exploding Gradients

Attached question

What is the smallest example that makes RNNs and Vanishing/Exploding Gradients click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action
Local action draft

This draft stays in this browser, attached to the selected learning item.

No local draft saved.
Evidence to inspect
  • References to inspect: attached references on this page.
  • Definition, prerequisite, and contrast concept links
  • The equation or runnable code that makes the concept operational
  • One demo state that shows the invariant instead of a slogan
What would resolve this
  • The learner can state the mechanism in their own words
  • The learner can name the prerequisite that would repair confusion
  • The learner can predict how the mechanism changes under one perturbation
Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - RNNs and Vanishing/Exploding Gradients Selected item key: recorded for copy. Context: Machine Learning Page anchor: recorded for copy. Open question: What is the smallest example that makes RNNs and Vanishing/Exploding Gradients click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context
concept/concept-notebook/machine-learning/rnns-vanishing-exploding-gradients concept:machine-learning/rnns-vanishing-exploding-gradients