A residual connection adds a layer’s output to its input instead of replacing the input. Residual connections made it practical to train networks with more than a hundred layers: He et al. trained a 152-layer network on ImageNet and later a 1001-layer network on CIFAR-10 (2016).
A newer notebook covers this concept. This older page keeps its summary, math and sources.
Concept 41 of 100Residual ConnectionsCore Training · Study phase 2: Architecture fundamentals
Question
Why is it useful to see a residual network as a perturbation of the identity, with each block making a small correction rather than a full transformation?
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you use the demo in the newer notebook, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the notebook demo shows with the key equation here and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueContinue in the newer notebook.
The newer notebook covers the same concept and has an interactive demo.
Why It Matters for Modern Models
A residual connection adds a layer’s output to its input instead of replacing the input. Residual connections made it practical to train networks with more than a hundred layers: He et al. trained a 152-layer network on ImageNet and later a 1001-layer network on CIFAR-10 (2016).
The residual-stream view is central to mechanistic interpretability: each attention head and MLP reads from and writes to one shared vector, so their contributions add up and can be studied one at a time.
The logit lens depends on this structure: because every layer adds to the same vector, intermediate states can be decoded with the model's final output layer.
Questions to Think About
In what sense does a deep residual network behave like an ensemble of many shallower paths, and what did deleting single blocks at test time show (Veit et al., 2016)?
How does placing layer normalization before each sublayer (pre-norm) rather than after the residual addition (post-norm) change gradient flow, and why can pre-norm transformers train without learning-rate warmup (Xiong et al., 2020)?
Interactive Demo
The interactive demo for this concept is in the newer notebook.
If you prefer intuition first, try the demo in the newer notebook, then come back for the math.
Key Equation
y=x+F(x)
A residual block adds a learned correction to its input instead of replacing it:
y=x+F(x,{Wi})
In a transformer, every attention and MLP sublayer adds its output to a shared vector called the residual stream. Leaving out layer normalization:
h′=h(l)+Attn(h(l)),h(l+1)=h′+MLP(h′)
Gradient flow. Writing each block as h(l)=h(l−1)+F(l)(h(l−1)), the chain rule gives
∂h(0)∂L=∂h(L)∂Ll=L∏1(I+∂h(l−1)∂F(l))
Expanding the product gives the identity plus terms that pass through one or more blocks. The identity term carries the gradient from the loss to every layer directly, so the gradient is unlikely to vanish even in very deep networks (He et al., 2016, "Identity Mappings").