Backpropagation

For a simple feed-forward network, write activations as

for layers $\ell=1,\dots,K$. For one training example, use column-vector shapes:

Let the scalar loss be

Backpropagation tracks sensitivities flowing backward from the loss. Define

For the final layer,

If the output layer is the identity, then $h_K=z_K$ and $\delta_K=\nabla_{h_K}J$. For squared error $J=\frac12\|\hat y-y\|^2$, this gives $\delta_K=\hat y-y$.

For earlier layers, the chain rule gives

Read the two factors separately. $W_{\ell+1}^{\mathsf T}\delta_{\ell+1}=\partial J/\partial h_\ell$ is the signal layer $\ell$ receives from downstream: how the loss changes with its activations. $\phi'(z_\ell)$ is the local derivative: how much those activations change with their pre-activations, computed from layer $\ell$ alone. The product $\odot$ is taken unit by unit.

Once $\delta_\ell$ is known, the parameter gradients are local:

This is the core efficiency: the same downstream sensitivity $\delta_\ell$ is reused to compute gradients for both parameters and earlier activations. For a scalar loss with many parameters, one backward pass computes the full gradient needed by gradient descent.

For a batch, the same local rules apply per example, then weight and bias gradients are summed or averaged across the batch dimension.

Worked pass: the notebook network

This is the network in the figure and in the Code below. It has $n_0=2$ inputs, $n_1=3$ tanh hidden units and one identity output, so $K=2$, and the loss is $J=\tfrac12(\hat y-y)^2$. Units and inputs are counted from 1: $W_1[i,j]$ is the weight from input $x_j$ into hidden unit $i$, called H$i$. The givens, with their shapes:

The forward pass keeps every value the backward pass will reuse:

The output is the identity, so $\delta_2=\hat y-y=-0.122215$. It is negative: raising $\hat y$ would lower the loss. Each hidden unit then gets its two factors and their product:

Hidden unit 1, step by step. H1 reaches the loss by one route, $z_{1,1}\to h_{1,1}\to\hat y\to J$, through the readout weight $W_2[1]=0.25$. So the signal it receives is $\partial J/\partial h_{1,1}=0.25\times(-0.122215)=-0.030554$: raising $h_{1,1}$ would lift $\hat y$ toward the target and lower the loss. Its local derivative ignores everything downstream: $1-0.500520^2=0.749480$, because tanh is still fairly steep at $z=0.55$. The delta is the product, $\delta_{1,1}=-0.030554\times0.749480=-0.022899$.

Now the weights. H1's pre-activation is $z_{1,1}=W_1[1,1]\,x_1+W_1[1,2]\,x_2+b_1[1]$, so $\partial z_{1,1}/\partial W_1[1,j]=x_j$: each weight's gradient is the delta times the input that weight multiplied. Row 1 of the gradient is

The signs carry meaning. The first entry is negative because $x_1$ is positive, so gradient descent increases $W_1[1,1]$. The second is positive because $x_2=-1$ flips it, so descent decreases $W_1[1,2]$. For this input both moves raise $z_{1,1}$, which lifts $\hat y$ toward the target through the positive readout weight.

H2 and H3. H2's readout weight is negative, so its received signal is positive, $0.018332$: raising $h_{1,2}$ would push $\hat y$ further below the target. Its local derivative equals H1's because $h_{1,2}=-h_{1,1}$ and squaring removes the sign. H3 follows the same two steps with its own numbers: received $-0.024443$, local derivative $0.786448$ at $z=-0.5$, delta $-0.019223$. Its gradient row is left for you to finish in the practice below.

The same deltas give $\partial J/\partial b_1=\delta_1$, the readout gradient $\partial J/\partial W_2=\delta_2h_1^{\mathsf T}=[-0.061171,\ 0.061171,\ 0.056478]$ and the signal passed to the input, $\partial J/\partial x=W_1^{\mathsf T}\delta_1=[-0.019784,\ 0.000464]^{\mathsf T}$. You can check any entry by nudging one parameter: the central difference $\big(J(\theta+\varepsilon)-J(\theta-\varepsilon)\big)/2\varepsilon$ with $\varepsilon=10^{-6}$ matches it.

A weight used in two places

In this network each weight is used once. When a value is used by several later operations, as a weight tied across positions or an input feeding several units is, each use contributes its own term, and the terms add; Baydin et al. (2018, Section 3.2) show the same accumulation for an input that reaches the output through two intermediate values. A two-branch graph makes it concrete:

with $\bar v=\partial L/\partial v$, so $\bar y=y-t$, $\bar a=\bar y\,b$ and $\bar b=\bar y\,a$. Take $s=1$, $w=3$, $x=2$ and $t=20$. Forward, $a=6$, $b=5$, $y=30$ and $L=50$; backward, $\bar y=10$, $\bar a=50$ and $\bar b=60$. Then $dL/dw=100+60=160$ and $dL/dx=150+60=210$; keeping only one path would give the derivative of a different graph. With $s=-1$ and $t=1$ instead, $\bar y=5$, $\bar a=5$ and $\bar b=30$, so $dL/dw=10+30=40$ while $dL/dx=15-30=-15$: the two paths to $x$ now partly cancel. Both the branch and the target changed between the two cases, so they do not compare one variable; the rule of adding every path's contribution is the same in both. The practice below works both cases.

import numpy as np

# The notebook network: separate numbers from the interactive Cases A/B/C.
# Shapes (column vectors):
# x: (2, 1), W1: (3, 2), b1: (3, 1)
# W2: (1, 3), b2: (1, 1), y: (1, 1)
x = np.array([[0.5], [-1.0]])
y = np.array([[0.25]])

W1 = np.array([[0.6, -0.2],
               [-0.3, 0.4],
               [0.1, 0.5]])
b1 = np.array([[0.05], [0.0], [-0.05]])

W2 = np.array([[0.25, -0.15, 0.2]])
b2 = np.array([[0.02]])

# Forward pass: store what the backward pass reuses
z1 = W1 @ x + b1            # (3, 1) pre-activations
h1 = np.tanh(z1)            # (3, 1) hidden activations
z2 = W2 @ h1 + b2           # (1, 1)
pred = z2                   # identity output, y-hat
loss = 0.5 * float(np.sum((pred - y) ** 2))

# Backward pass
delta2 = pred - y           # dJ/dz2, (1, 1)
dW2 = delta2 @ h1.T         # (1, 3)
db2 = delta2

dh1 = W2.T @ delta2         # received signal dJ/dh1, (3, 1)
phi_prime = 1 - h1 ** 2     # local derivative tanh'(z1), (3, 1)
delta1 = dh1 * phi_prime    # dJ/dz1 = received * local, (3, 1)
dW1 = delta1 @ x.T          # (3, 1) @ (1, 2) -> (3, 2)
db1 = delta1
dx = W1.T @ delta1          # input cotangent, (2, 1)


def show(name, value):
    print(f"{name:<10}", np.round(np.ravel(value), 6))


print("loss:", round(loss, 6))
for name, value in [("z1", z1), ("h1", h1), ("pred", pred), ("delta2", delta2),
                    ("dh1", dh1), ("phi_prime", phi_prime), ("delta1", delta1)]:
    show(name, value)
show("dW1 row 1", dW1[0])

# Row 3 (Python index 2): hidden unit 3's delta times the input row x^T
row3 = delta1[2, 0] * x.T   # (1, 2)
show("dW1 row 3", row3)
assert np.allclose(row3, dW1[2])

show("dW2", dW2)
show("dx", dx)
print("hidden signal shape:", delta1.shape)
print("first-layer gradient shape:", dW1.shape)
print("input cotangent shape:", dx.shape)

It prints the same named values as the worked pass, in this order: loss, z1, h1, pred for $\hat y$, delta2, dh1 for the received signal $W_2^{\mathsf T}\delta_2$, phi_prime for the local derivative $1-h_1^2$, delta1, row 1 of dW1, H3's row delta1[2, 0] * x.T checked against dW1[2], then dW2, dx and the three shapes. Python counts from 0, so H3 is index 2. That row answers the H3 exercise above, so try it first.

This tiny network has one hidden layer, but it shows the recurrence: delta2 moves through W2.T, becomes delta1 after the tanh derivative, and then produces both dW1 and dx. A deeper network repeats those lines once per layer, from the loss back to the input.

As an algorithm, backpropagation is settled: it is the chain rule, evaluated from the loss backwards so that each intermediate result is computed once. Two questions around it are not.

Brains learn, but a neuron has no obvious way to send an exact error backwards through the same synapses it used going forward. Whether the brain approximates backpropagation, and how, is argued over (Lillicrap et al., 2020, “Backpropagation and the brain”). One surprise from that line of work: fixed random backward weights, in place of the transposed forward ones, still let a network learn (Lillicrap et al., 2016).

The other question is why following these gradients works so well at all. The loss of a deep network is far from a simple bowl, yet small steps downhill reliably find good solutions, and a full explanation is still missing.