Backpropagation computes the gradient of a scalar loss with respect to each parameter by applying the chain rule backward through the network. These gradients are used to train nearly every modern neural network.
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Backpropagation computes the gradient of a scalar loss with respect to each parameter by applying the chain rule backward through the network. These gradients are used to train nearly every modern neural network.
Storing activations for the backward pass is a main memory cost of training, which is why activation checkpointing (recomputing activations instead of storing them) exists, and why large batches are often split into smaller ones whose gradients are accumulated.
A training step costs roughly three forward passes and must also keep activations and optimizer state in memory, a large part of why training a model costs far more than running it.
Questions to Think About
PyTorch records the computation graph as the forward pass runs; why does torch.no_grad() therefore save memory as well as computation?
How do repeated products of Jacobians make gradients vanish or explode, and how does the identity path of a residual connection reduce the problem?
Core Math
Key Equation
δℓ=δℓ+1⋅∂hℓ∂hℓ+1
Backpropagation computes the gradient of a scalar loss with respect to every parameter by applying the chain rule from the output back toward the input. It is the special case of reverse-mode automatic differentiation used to train neural networks.
For a network with layers hℓ=fℓ(hℓ−1;θℓ), ℓ=1,…,L, and loss L(hL):
∂θℓ∂L=∂hL∂L⋅∂hL−1∂hL⋯∂hℓ∂hℓ+1⋅∂θℓ∂hℓ
The backward pass evaluates this product from the left, passing a row vector of sensitivities down the network:
δℓ=∂hℓ∂L=δℓ+1⋅∂hℓ∂hℓ+1
Each step is a vector–Jacobian product, so no full Jacobian matrix is ever formed.
Cost. Reverse mode returns the whole gradient for a small constant multiple of the cost of evaluating the loss, however many parameters there are (Baydin et al., 2018). For the matrix multiplications that dominate neural networks, the backward pass costs about twice the forward pass, because each layer needs gradients with respect to both its input and its weights. A training step therefore costs about three forward passes, which is where the common estimate of 6N floating-point operations per token for a model with N parameters comes from (Kaplan et al., 2020).
Memory. The backward pass needs the intermediate activations h1,…,hL from the forward pass, so they must be stored or recomputed.
Key Papers and Books
Learning representations by back-propagating errors