Machine Learning

LSTM and GRU Gates

Read LSTM and GRU equations as learned memory routes: carry old state, write a candidate, reset irrelevant context, and expose the result when the sequence asks for it.

status: reviewimportance: criticaldifficulty 4/5math: graduateread: 22mlive demo

Back to Machine Learning Next: Seq2seq and Encoder-Decoder Attention (review)

Concept Structure

LSTM and GRU Gates

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

2prerequisites

1next concepts

3related links

Learner Contract

What this page should let you do.

You are here becauseRead LSTM and GRU equations as learned memory routes: carry old state, write a candidate, reset irrelevant context, and expose the result when the sequence asks for it.

This Machine Learning concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

Before thisRNNs and Vanishing/Exploding Gradients (review)

2 prerequisites listed; refresh them before leaning on the math or code.

RNNs and Vanishing/Exploding Gradients (review)Activation Functions and Gradient Flow (review)

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextSeq2seq and Encoder-Decoder Attention (review)

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to Seq2seq and Encoder-Decoder Attention (review)

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.

Claims1/1 reviewed

Sources5 cited

Codeattached

Demolive

Reviewed2026-07-02

Updatedpage 2026-07-02

Learning item flow

4/4 sections readyAsk about this Research room

ConceptLSTM and GRU GatesMachine Learning EquationLSTM and GRU Gates equation 1Exact equation CodeLSTM and GRU Gates runnable code 1Exact runnable code DemoLSTM and GRU Gates interactive demoInteractive demo ClaimLSTM and GRU gates are sigmoid-controlled elementwise routes that let...Exact claim check SourceDive into Deep Learning: Long Short-Term MemoryExact source

ConceptLSTM and GRU GatesMachine Learning

5 sources attachedLocal snapshot ready

concept:machine-learning/lstm-gru-gates

Codenearby Predictbefore reveal Roomitem handoff

01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

A plain RNN tries to remember by repeatedly replacing one hidden state with the next. That is why the previous page cared so much about repeated Jacobian products: long-range credit assignment has to survive many recurrent transitions.

LSTM and GRU units change the question. Instead of asking "what should the next hidden state be?" they ask smaller routing questions at every time step:

Should the old state keep flowing?
Should this new candidate be written?
Should old context be reset before making the candidate?
Should the memory be visible to downstream layers right now?

The gates are not little symbolic if-statements. They are learned sigmoid vectors with values between $0$ and $1$ . A coordinate near $1$ opens a route. A coordinate near $0$ mostly closes it. Different coordinates can make different decisions at the same time.

The important mental picture is a memory highway with controlled side doors. In an LSTM, the cell state $c_t$ has an additive update:

c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t.

One route carries old memory; another writes new candidate memory. The output gate then decides how much of the internal cell becomes the exposed hidden state.

In a GRU, the hidden state is also the memory, and the update gate interpolates between old state and candidate:

h_t = z_t \odot h_{t-1} + (1-z_t)\odot \tilde{h}_t.

When $z_t$ is near $1$ , the old state mostly persists. When $z_t$ is near $0$ , the candidate mostly replaces it. The reset gate shapes the candidate by deciding how much old state the candidate computation can see.

So gates are not magic memory. They are learned routes that make useful memory easier to represent and train. A gate can keep the wrong thing, overwrite the right thing, or expose memory at the wrong time. The architecture gives the optimizer better handles; the task and training still decide what the handles learn.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt

Equation 1 $\begin{aligned} i_t &= \sigma(W_{xi}x_t + W_{hi}h_{t-1}+b_i),\\ f_t &= \sigma(W_{xf}x_t + W_{hf}h_{t-1}+b_f),\\ o_t &= \sigma(W_{xo}x_t + W_{ho}h_{t-1}+b_o). \end{aligned}$ Equation 2 $\tilde{c}_t = \tanh(W_{xc}x_t + W_{hc}h_{t-1}+b_c).$

Let $x_t\in\mathbb R^d$ be the input at time $t$ . Let $\sigma(u)=1/(1+e^{-u})$ be the sigmoid function and let $\odot$ denote elementwise multiplication.

For an LSTM, the hidden state is $h_t\in\mathbb R^m$ and the internal cell state is $c_t\in\mathbb R^m$ . The gates are

\begin{aligned} i_t &= \sigma(W_{xi}x_t + W_{hi}h_{t-1}+b_i),\\ f_t &= \sigma(W_{xf}x_t + W_{hf}h_{t-1}+b_f),\\ o_t &= \sigma(W_{xo}x_t + W_{ho}h_{t-1}+b_o). \end{aligned}

The candidate cell write is

\tilde{c}_t = \tanh(W_{xc}x_t + W_{hc}h_{t-1}+b_c).

The cell and hidden updates are

\begin{aligned} c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{c}_t,\\ h_t &= o_t \odot \tanh(c_t). \end{aligned}

Read those equations as three route decisions:

$f_t$ is the carry route for old cell memory.
$i_t$ is the write route for the candidate.
$o_t$ is the exposure route from internal memory to hidden state.

If you temporarily hold the gates and candidate fixed while looking at the direct cell-state path, then one coordinate obeys

\frac{\partial c_t}{\partial c_{t-1}} = f_t.

Across many steps, that direct path contributes a product of forget gates. If the relevant $f_t$ values stay near $1$ , the cell gives information and gradients a much better corridor than repeated full hidden-state replacement. If the forget gates close or the input gates keep writing distractors, the memory can still be lost.

For a GRU, there is no separate cell state. The hidden state $h_t\in\mathbb R^m$ is the memory seen by the next recurrent step. Using the convention from D2L, the reset and update gates are

\begin{aligned} r_t &= \sigma(W_{xr}x_t + W_{hr}h_{t-1}+b_r),\\ z_t &= \sigma(W_{xz}x_t + W_{hz}h_{t-1}+b_z). \end{aligned}

The candidate is

\tilde{h}_t = \tanh(W_{xh}x_t + W_{hh}(r_t\odot h_{t-1}) + b_h),

and the final hidden update is

h_t = z_t \odot h_{t-1} + (1-z_t)\odot\tilde{h}_t.

Here $r_t$ decides how much old state the candidate can consult. The update gate $z_t$ decides how much old state carries directly into $h_t$ . Some papers and libraries use the opposite interpolation convention, so always check whether "update" means "keep old" or "write new" in the source you are reading.

The mathematical bridge from RNN gradient pathologies is now visible:

\text{plain RNN: repeated nonlinear replacement} \qquad\longrightarrow\qquad \text{gated RNN: direct carry plus controlled write}.

That bridge is why LSTMs and GRUs were such important pre-transformer sequence models.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt

Runnable code 1import numpy as np tokens = ["KEY", "noise", "pause", "DISTRACTOR", "QUERY"] candidates = np....python

import numpy as np

tokens = ["KEY", "noise", "pause", "DISTRACTOR", "QUERY"]
candidates = np.array([0.9, 0.05, -0.05, -0.9, 0.0])

presets = {
    "keep": (
        [0.00, 0.96, 0.96, 0.97, 0.98],  # forget/carry
        [0.95, 0.04, 0.05, 0.03, 0.02],  # input/write
        [0.08, 0.08, 0.10, 0.12, 0.28],  # output/expose
    ),
    "overwrite": (
        [0.00, 0.92, 0.90, 0.16, 0.90],
        [0.95, 0.08, 0.08, 0.88, 0.12],
        [0.08, 0.08, 0.10, 0.18, 0.35],
    ),
    "expose": (
        [0.00, 0.96, 0.97, 0.97, 0.98],
        [0.95, 0.04, 0.03, 0.03, 0.02],
        [0.08, 0.08, 0.10, 0.18, 0.92],
    ),
}

def run_lstm_route(name):
    c = 0.0
    rows = []
    f_gate, i_gate, o_gate = presets[name]
    for token, cand, f, i, o in zip(tokens, candidates, f_gate, i_gate, o_gate):
        c = f * c + i * cand
        h = o * np.tanh(c)
        rows.append((token, round(f, 2), round(i, 2), round(o, 2), round(c, 3), round(h, 3)))
    return rows

for name in presets:
    print("\n", name)
    for row in run_lstm_route(name):
        print(row)

The numbers are hand-set to isolate the mechanism. A trained LSTM learns the gates from data, and each gate is a vector, not a scalar. Still, the code makes the central object impossible to miss: the final cell is a sum of a carried old state and gated candidate writes.

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the Gate Routing Lab to turn the equations into a hidden-state routing prediction.

Before reveal, choose an LSTM or GRU, inspect the token sequence, and commit to one outcome:

Kept: the target memory survives the distractor.
Overwritten: the distractor replaces the target memory.
Exposed: the target memory becomes visible to the readout at the query step.

After reveal, inspect the gate values, state trace, route summary, and caveat strip. The lab is a finite deterministic witness, not a proof that LSTMs or GRUs always solve long-range dependency tasks. Real gates are learned, vector-valued, input-dependent, and trained through the same loss surface as the rest of the network.

Live Concept Demo

Explore LSTM and GRU Gates

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5graduatecode-aligned

Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what LSTM and GRU Gates should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Read LSTM and GRU equations as learned memory routes: carry old state, write a candidate, reset irrelevant context, and expose the result when the sequence asks for it.

Prediction open01 / Intuition

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change LSTM and GRU Gates should make visible.

Visual Inquiry

Make the image answer a mathematical question

Read LSTM and GRU equations as learned memory routes: carry old state, write a candidate, reset irrelevant context, and expose the result when the sequence asks for it.

4/4 stages readyLive demo connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make LSTM and GRU Gates easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

book · 2026Dive into Deep Learning: Long Short-Term MemoryZhang, Lipton, Li, and Smola

Defines LSTM input, forget, output gates, candidate cell, cell-state update, hidden-state exposure, and the constant-memory special case.

Open source

book · 2026Dive into Deep Learning: Gated Recurrent UnitsZhang, Lipton, Li, and Smola

Defines GRU reset/update gates, candidate hidden state, and update-gate interpolation between old state and candidate.

Open source

course-notes · 2019CS224N: Language Models, RNN, GRU and LSTMStanford CS224N

Course source for GRU/LSTM equations and intuition about persistent memory, new memory, reset, update, forget, input, and exposure gates.

Open source

paper · 1997Long Short-Term MemoryHochreiter and Schmidhuber

Primary source for LSTM memory cells, constant error flow, and multiplicative input/output gates controlling access to cell contents.

Open source

paper · 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine TranslationCho, van Merrienboer, Gulcehre, Bahdanau, Bougares, Schwenk, and Bengio

Primary source for the GRU-style reset and update gates, hidden-state interpolation, and adaptive memory timescales.

Open source

Claim Review

Read LSTM and GRU equations as learned memory routes: carry old state, write a candidate, reset irrelevant context, and expose the result when the sequence asks for it.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources5 references

d2l-2026-lstm, d2l-2026-gru, cs224n-2019-rnn-gru-lstm-notes, hochreiter-schmidhuber-1997-lstm, cho-2014-rnn-encoder-decoder

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedLSTM and GRU gates are sigmoid-controlled elementwise routes that let learned recurrent units keep, overwrite/reset, and expose memory through additive or interpolated state updates; this alleviates long-range training difficulty but does not guarantee perfect memory.Claim metadata: source checked

D2L supports the LSTM gate/cell equations, the GRU reset/update equations, and the keep/write/expose interpretations; CS224N supports the same course-level equations and gate intuitions; Hochreiter-Schmidhuber ground the LSTM memory-cell and constant-error-flow motivation; Cho et al. ground the GRU reset/update gates and adaptive remember/forget behavior.

Sources: Dive into Deep Learning: Long Short-Term Memory, Dive into Deep Learning: Gated Recurrent Units, CS224N: Language Models, RNN, GRU and LSTM, Long Short-Term Memory, Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine TranslationFinite deterministic gate-routing witness only; real gates are vector-valued, learned from data, coupled through nonlinear candidates, and affected by training details.A bounded review summary is present; still check caveats and exact reference scope.

Checked D2L LSTM/GRU chapters, Stanford CS224N RNN notes, Hochreiter-Schmidhuber 1997, and Cho et al. 2014 for gate equations, controlled memory routing, constant/direct carry paths, update/reset behavior, and long-range-memory caveats. The demo is a finite scalar/vector routing witness, not a trained architecture benchmark. GPT Pro/Oracle publication critique remains pending because 127.0.0.1:51672 refused connection.

Reviewer: codex-local-primary-source-audit; reviewed 2026-07-02

source-span-d2l-2026-lstm source-span-d2l-2026-gru source-span-cs224n-2019-rnn-gru-lstm-notes Equation 1 Equation 2 Runnable code 1 Interactive demo

Source support candidates

book 2026Dive into Deep Learning: Long Short-Term Memory

Defines LSTM input, forget, output gates, candidate cell, cell-state update, hidden-state exposure, and the constant-memory special case.

book 2026Dive into Deep Learning: Gated Recurrent Units

Defines GRU reset/update gates, candidate hidden state, and update-gate interpolation between old state and candidate.

course-notes 2019CS224N: Language Models, RNN, GRU and LSTM

Course source for GRU/LSTM equations and intuition about persistent memory, new memory, reset, update, forget, input, and exposure gates.

paper 1997Long Short-Term Memory

Primary source for LSTM memory cells, constant error flow, and multiplicative input/output gates controlling access to cell contents.

Mechanism checks

Equation 1

\begin{aligned} i_t &= \sigma(W_{xi}x_t + W_{hi}h_{t-1}+b_i),\\ f_t &= \sigma(W_{xf}x_t + W_{hf}h_{t-1}+b_f),\\ o_t &= \sigma(W_{xo}x_t + W_{ho}h_{t-1}+b_o). \end{aligned}

Equation 2

\tilde{c}_t = \tanh(W_{xc}x_t + W_{hc}h_{t-1}+b_c).

Runnable code 1import numpy as np tokens = ["KEY", "noise", "pause", "DISTRACTOR", "QUERY"] candidates = np....Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Read LSTM and GRU equations as learned memory routes: carry old state, write a candidate, reset irrelevant context, and expose the result when the sequence asks for it.

Readiness0/3 checks ready

Predict

Before touching the demo, predict one visible change that should happen in LSTM and GRU Gates.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Local checks

Claim

A concrete answer is on the canvas.

Mechanism

The answer names why the claim should hold.

Bridge

It touches the page context or a neighboring idea.

Misconception check

Grounded research drawerClose

ConceptLSTM and GRU GatesMachine Learning

Runnable code comparisonLSTM and GRU Gates runnable code 1tokens = ["KEY", "noise", "pause", "DISTRACTOR", "QUERY"]Prediction before revealLSTM and GRU Gates interactive demoManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes LSTM and GRU Gates click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptMachine Learning

LSTM and GRU Gates

Attached question

What is the smallest example that makes LSTM and GRU Gates click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays in this browser, attached to the selected learning item.

Draft noteNext action

No local draft saved.

Evidence to inspect

References to inspect: attached references on this page.
Definition, prerequisite, and contrast concept links
The equation or runnable code that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - LSTM and GRU Gates Selected item key: recorded for copy. Context: Machine Learning Page anchor: recorded for copy. Open question: What is the smallest example that makes LSTM and GRU Gates click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context

concept/concept-notebook/machine-learning/lstm-gru-gates
concept:machine-learning/lstm-gru-gates

Learning Map

Before / Now / Try / Next

BeforeRNNs and Vanishing/Exploding Gradients (review)

NowIntuition -> Math -> Code -> Demo

TryManipulate one control and predict the visible change.

NextSeq2seq and Encoder-Decoder Attention (review)

Intuitionready
Mathready
Codeready
Interactive Demoready

AI Companion

Ask beside the selected object

Read LSTM and GRU equations as learned memory routes: carry old state, write a candidate, reset irrelevant context, and expose the result when the sequence asks for it.

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Machine Learning concept. Learning surface: LSTM and GRU Gates. What this page says: Read LSTM and GRU equations as learned memory routes: carry old state, write a candidate, reset irrelevant context, and expose the result when the sequence asks for it. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Domain

Machine Learning

machine-learningdeep-learningrecurrent-neural-networkssequence-modelinggates

Prerequisites

RNNs and Vanishing/Exploding Gradients (review)Activation Functions and Gradient Flow (review)

Leads To

Seq2seq and Encoder-Decoder Attention (review)

Gradient Clipping & Explosion Prevention (review)Residual Connections & Skip Connections (review)BatchNorm, LayerNorm, and RMSNorm (review)

Within this domain