How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you move a control in the demo, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the demo shows with the key equation and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Probes are a standard way to test what information a model's hidden states contain and at which layer it becomes linearly readable.
A probe shows that information can be decoded, not that the model uses it; control tasks (Hewitt & Liang, 2019) and causal interventions are needed to tell the two apart.
Questions to Think About
How does information move from layer to layer in a large language model?
Interactive Demo
Core Math
If you prefer intuition first, try the demo above, then come back for the math.
Key Equation
y^=Whℓ(x)+b
Given the representation hℓ(x) at layer ℓ, keep the network frozen and train a linear probe:
y^=Whℓ(x)+b
(or a softmax over Whℓ(x)) on a supervised task such as part-of-speech tags or syntactic relations. The probe's accuracy estimates how linearly decodable that information is at layer ℓ.
Tenney et al. (2019) found that BERT's layers roughly follow the order of the classical NLP pipeline: part-of-speech tags, then parsing, then named entities and semantic roles, then coreference.
Key Papers
Understanding Intermediate Layers using Linear Classifier Probes
Which limitations does it address, or leave to a later idea?
Causal circuits: a limitation of Probing that Circuit Discovery addresses. Open Circuit Discovery
Probing is correlational: it shows information is decodable, not that it drives behavior. Circuit discovery was invented to identify causal subgraphs via interventions/patching and attribution.
Where does it break down?
Non-causal probes: where Probing breaks down, taken up in Superposition. Open Superposition
With superposition, probes can "find" information in activations that isn't a clean, causal feature the model uses. High probe accuracy can overstate mechanistic understanding when features are entangled.
Features overlap: where Superposition breaks down, taken up on this page. Open Superposition
Under superposition a layer stores more features than it has dimensions, as directions that are only nearly orthogonal. A linear probe for one feature then also picks up small contributions from the others that overlap with it, so its accuracy is limited by this interference, and the direction it finds need not be a single feature the model uses.
Where else does the same idea appear?
Linear lens: an analogy between Probing and NTK. Open NTK
NTK emphasizes what happens when learning is effectively linearized (features don't move much). Linear probing tests what a fixed representation already makes linearly accessible—both separate "linear extractability" from "feature learning."
Which ideas are its mathematical duals?
Top-down vs bottom-up: Circuit Discovery is its mathematical dual. Open Circuit Discovery
Probing asks "what info is here?" (bottom-up). Circuit discovery asks "what computation uses this?" (top-down). Complementary interpretability approaches.