The logit lens applies a transformer’s output layer to its intermediate hidden states. It shows the token predictions those states produce at each layer, without additional training.
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
The logit lens applies a transformer’s output layer to its intermediate hidden states. It shows the token predictions those states produce at each layer, without additional training.
In GPT-2 it showed predictions forming gradually, with intermediate layers often already close to the final answer (nostalgebraist, 2020).
It is the starting point for trained decoders such as the tuned lens, which Belrose et al. (2023) report to be more predictive and reliable than the raw logit lens.
Questions to Think About
Why do some layers, and some models, give meaningless tokens under the logit lens even though they carry useful information?
What does a learned affine map per layer (the tuned lens) correct for, and why does it work better than the raw logit lens?
Core Math
Key Equation
logits(l)=WULNf(h(l))
The logit lens decodes an intermediate residual-stream state with the model's own output layer, as if the network ended at that layer:
logits(l)=WULNf(h(l))
where h(l) is the residual stream after layer l, LNf is the final layer normalization and WU is the unembedding matrix. The corresponding next-token distribution is
p(l)(token)=softmax(logits(l))
Following p(0)→p(L) shows how the prediction forms across layers. In GPT-2, nostalgebraist (2020) found that predictions move gradually toward the final output.
Tuned lens. Belrose et al. (2023) train a separate affine map for each layer before decoding,
logits(l)=WULNf(Alh(l)+bl)
which corrects for intermediate layers not representing information in exactly the same basis as the final layer.