Continuous Function

Calculus

Backpropagation

Backpropagation applies reverse-mode autodiff to neural networks so one scalar loss can train many parameters.

status: publishedimportance: criticaldifficulty 3/5math: undergraduateread: 15mlive demo
Back to CalculusNext: Gradient Descent
Editorial neural-network illustration of activations flowing forward and error signals propagating backward through layers.

Concept Structure

Backpropagation

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

1prerequisites
5next concepts
2related links

Learning map

Backpropagation
BeforeReverse-Mode Automatic DifferentiationNow4/4 sections readyTryManipulate one control and predict the visible change.NextGradient Descent

Object flow

4/4 sections readyAsk about thisResearch room
ConceptBackpropagationCalculusEquationBackpropagation equation 1Exact equation objectCodeBackpropagation code witness 1Exact code witnessDemoBackpropagation interactive demoVisualization objectClaimBackpropagation computes neural-network gradients by storing needed f...Exact claim checkSourceLearning representations by back-propagating errorsExact source object
ConceptBackpropagationCalculus
2 sources attachedLocal snapshot ready
concept:calculus/backpropagation
Codewitness nearbyPredictbefore revealRoomobject handoff

Conceptual Bridge

What should feel connected as you move through this page.

Carry inReverse-Mode Automatic Differentiation

Bring the mental model from Reverse-Mode Automatic Differentiation; this page will reuse it instead of restarting from zero.

Work hereBackpropagation

Backpropagation applies reverse-mode autodiff to neural networks so one scalar loss can train many parameters.

Carry outGradient Descent

The next edge should feel earned: use the demo prediction here before following Gradient Descent.

Test the linkManipulate one control and predict the visible change.Then continue to Gradient Descent
01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.
01

01

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

Backpropagation is reverse-mode autodiff specialized to neural networks.

A neural network is a layered computation graph. The forward pass turns inputs into predictions and a loss. The backward pass asks each layer a local question: "if the loss is sensitive to my output, how sensitive is it to my inputs and parameters?"

The answer is passed backward layer by layer. Each parameter receives a gradient, and an optimizer uses those gradients to update the model. Backpropagation is not magic pattern recognition; it is the chain rule organized so every local derivative is reused efficiently.

02

02

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt
Equation 1h_0=x,\qquad z_\ell=W_\ell h_{\ell-1}+b_\ell,\qquad h_\ell=\phi(z_\ell),Equation 2h_{\ell-1}\in\mathbb{R}^{n_{\ell-1}},\quad W_\ell\in\mathbb{R}^{n_\ell\times n_{\ell-1}},\qua...

For a simple feed-forward network, write activations as

h0=x,zℓ=Wℓhℓ−1+bℓ,hℓ=ϕ(zℓ),h_0=x,\qquad z_\ell=W_\ell h_{\ell-1}+b_\ell,\qquad h_\ell=\phi(z_\ell),

for layers ℓ=1,…,K\ell=1,\dots,K. For one training example, use column-vector shapes:

hℓ−1∈Rnℓ−1,Wℓ∈Rnℓ×nℓ−1,bℓ,zℓ,hℓ∈Rnℓ.h_{\ell-1}\in\mathbb{R}^{n_{\ell-1}},\quad W_\ell\in\mathbb{R}^{n_\ell\times n_{\ell-1}},\quad b_\ell,z_\ell,h_\ell\in\mathbb{R}^{n_\ell}.

Let the scalar loss be

J=ℓ(hK,y).J=\ell(h_K, y).

Backpropagation tracks sensitivities flowing backward from the loss. Define

δℓ=∂J∂zℓ∈Rnℓ.\delta_\ell = \frac{\partial J}{\partial z_\ell}\in\mathbb{R}^{n_\ell}.

For the final layer,

δK=∇hKJ⊙ϕ′(zK).\delta_K = \nabla_{h_K}J \odot \phi'(z_K).

If the output layer is the identity, then hK=zKh_K=z_K and δK=∇hKJ\delta_K=\nabla_{h_K}J. For squared error J=12∥y^−y∥2J=\frac12\|\hat y-y\|^2, this gives δK=y^−y\delta_K=\hat y-y.

For earlier layers, the chain rule gives

δℓ=(Wℓ+1Tδℓ+1)⊙ϕ′(zℓ).\delta_\ell = (W_{\ell+1}^{\mathsf T}\delta_{\ell+1}) \odot \phi'(z_\ell).

Once δℓ\delta_\ell is known, the parameter gradients are local:

∂J∂Wℓ=δℓhℓ−1T,∂J∂bℓ=δℓ.\frac{\partial J}{\partial W_\ell} = \delta_\ell h_{\ell-1}^{\mathsf T},\qquad \frac{\partial J}{\partial b_\ell}=\delta_\ell.

This is the core efficiency: the same downstream sensitivity δℓ\delta_\ell is reused to compute gradients for both parameters and earlier activations. For a scalar loss with many parameters, one backward pass computes the full gradient needed by gradient descent.

For a batch, the same local rules apply per example, then weight and bias gradients are summed or averaged across the batch dimension.

03

03

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt
Code witness 1import numpy as np # Separate numbers from the interactive demo. # Shapes: # x: (2, 1), W1: (...python
import numpy as np

# Separate numbers from the interactive demo.
# Shapes:
# x: (2, 1), W1: (3, 2), b1: (3, 1)
# W2: (1, 3), b2: (1, 1), y: (1, 1)
x = np.array([[0.5], [-1.0]])
y = np.array([[0.25]])

W1 = np.array([[0.6, -0.2],
               [-0.3, 0.4],
               [0.1, 0.5]])
b1 = np.array([[0.05], [0.0], [-0.05]])

W2 = np.array([[0.25, -0.15, 0.2]])
b2 = np.array([[0.02]])

# Forward pass
z1 = W1 @ x + b1
h1 = np.tanh(z1)
z2 = W2 @ h1 + b2
pred = z2
loss = 0.5 * float(np.sum((pred - y) ** 2))

# Backward pass
delta2 = pred - y
dW2 = delta2 @ h1.T
db2 = delta2

dh1 = W2.T @ delta2
delta1 = dh1 * (1 - h1 ** 2)
dW1 = delta1 @ x.T
db1 = delta1
dx = W1.T @ delta1

print("loss:", round(loss, 4))
print("hidden signal shape:", delta1.shape)
print("first-layer gradient shape:", dW1.shape)
print("input cotangent shape:", dx.shape)

This tiny network has one hidden layer, but it shows the recurrence: delta2 moves through W2.T, becomes delta1 after the tanh derivative, and then produces both dW1 and dx. The interactive demo below uses a different concrete network and hides its backward values until you predict the hidden learning-signal path.

04

04

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the demo to step through one tiny two-layer network. First inspect the stored forward values, the target, and the readout weights. Before seeing the backward pass, predict which hidden unit will carry the strongest usable learning signal after the output error passes through the readout weights and local tanh gates.

Use Case A, Case B, and Case C as neutral graph states. Commit first, then reveal the hidden deltas, tanh gates, row-gradient norms, and update outcome.

Live Concept Demo

Explore Backpropagation

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 3/5undergraduatecode-aligned
Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Backpropagation should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Backpropagation applies reverse-mode autodiff to neural networks so one scalar loss can train many parameters.

Prediction open01 / Intuition
Editorial neural-network illustration of activations flowing forward and error signals propagating backward through layers.
Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Backpropagation should make visible.

Visual Inquiry

Make the image answer a mathematical question

Backpropagation applies reverse-mode autodiff to neural networks so one scalar loss can train many parameters.

4/4 stages readyLive demo connected
Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Backpropagation easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

paper · 1986Learning representations by back-propagating errorsRumelhart, Hinton, and Williams

Grounds backpropagation as error signals propagated through hidden units to adjust network weights.

Open source
paper · 2018Automatic differentiation in machine learning: a surveyBaydin et al.

Grounds the relationship between backpropagation, reverse-mode autodiff, and computation graphs.

Open source

Claim Review

Backpropagation applies reverse-mode autodiff to neural networks so one scalar loss can train many parameters.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources2 references

rumelhart-1986-backprop, baydin-2018-ad-survey

Witnesses4 local objects

Use equation, code, and demo objects to check whether the source support is operational.

Substantively reviewedBackpropagation computes neural-network gradients by storing needed forward-pass values, then applying the chain rule backward from a scalar loss: each layer receives a downstream sensitivity, combines it with local derivatives/VJPs, and reuses that signal to form parameter gradients in one reverse pass.Claim metadata: source checked

Rumelhart, Hinton, and Williams introduce back-propagating error signals for adjusting weights; Baydin et al. frame backpropagation as reverse-mode automatic differentiation on computation graphs for efficient derivative evaluation.

Sources: Learning representations by back-propagating errors, Automatic differentiation in machine learning: a surveyChecks the differentiable feed-forward scalar-loss mechanism, not optimizer choice, biological plausibility, recurrent/dynamic variants, full batching, nonsmooth primitives, or stability. Reverse-mode systems may save values directly or recover some through checkpointing.A bounded review summary is present; still check caveats and exact source scope.

Checked Rumelhart/Hinton/Williams for neural-network error back-propagation and weight adjustment, and Baydin et al. for mechanism: backprop is reverse-mode AD on a network objective, with forward intermediate/dependency recording, backward adjoint propagation, local chain-rule accumulation, VJP framing, output-adjoint seed 1, and one reverse pass computing the full scalar-objective gradient. Local math/code/demo instantiate the same feed-forward scalar-loss case with deltas, tanh gates, W^T delta, and dW/db.

Reviewer: codex+oracle; reviewed 2026-05-07
source-span-rumelhart-1986-backpropsource-span-baydin-2018-ad-surveymath-object-1code-witness-1interactive-demo

Source support candidates

paper 1986Learning representations by back-propagating errors

Grounds backpropagation as error signals propagated through hidden units to adjust network weights.

paper 2018Automatic differentiation in machine learning: a survey

Grounds the relationship between backpropagation, reverse-mode autodiff, and computation graphs.

Mechanism witnesses

Equation 1
h0=x,zℓ=Wℓhℓ−1+bℓ,hℓ=ϕ(zℓ),h_0=x,\qquad z_\ell=W_\ell h_{\ell-1}+b_\ell,\qquad h_\ell=\phi(z_\ell),
Equation 2
hℓ−1∈Rnℓ−1,Wℓ∈Rnℓ×nℓ−1,bℓ,zℓ,hℓ∈Rnℓ.h_{\ell-1}\in\mathbb{R}^{n_{\ell-1}},\quad W_\ell\in\mathbb{R}^{n_\ell\times n_{\ell-1}},\quad b_\ell,z_\ell,h_\ell\in\mathbb{R}^{n_\ell}.
Code witness 1import numpy as np # Separate numbers from the interactive demo. # Shapes: # x: (2, 1), W1: (...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Backpropagation applies reverse-mode autodiff to neural networks so one scalar loss can train many parameters.

Readiness0/3 checks ready
Predict

Before touching the demo, predict one visible change that should happen in Backpropagation.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Object research drawerClose
ConceptBackpropagationCalculus
Code witness comparisonBackpropagation code witness 1x = np.array([[0.5], [-1.0]])Prediction before revealBackpropagation interactive demoManipulate one control and predict the visible change.
Grounded room questionWhat is the smallest example that makes Backpropagation click without losing the math?Local snapshot ready

Research Room

Attach the question to an exact object

Pick the concept, equation, source, code witness, claim, misconception, or demo state before asking for help. The handoff stays grounded to that object.
Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptCalculus

Backpropagation

Anchored question

What is the smallest example that makes Backpropagation click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action
Local action draft

This draft stays locally in this browser for concept:calculus/backpropagation.

No local draft saved.
Evidence to inspect
  • Source ids to inspect: rumelhart-1986-backprop, baydin-2018-ad-survey
  • Definition, prerequisite, and contrast concept links
  • The equation or code witness that makes the concept operational
  • One demo state that shows the invariant instead of a slogan
What would resolve this
  • The learner can state the mechanism in their own words
  • The learner can name the prerequisite that would repair confusion
  • The learner can predict how the mechanism changes under one perturbation
Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Backpropagation Object key: concept:calculus/backpropagation Context: Calculus Anchor id: concept/concept-notebook/calculus/backpropagation Open question: What is the smallest example that makes Backpropagation click without losing the math? Evidence to inspect: - Source ids to inspect: rumelhart-1986-backprop, baydin-2018-ad-survey - Definition, prerequisite, and contrast concept links - The equation or code witness that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

Open source object
concept/concept-notebook/calculus/backpropagation concept:calculus/backpropagation