Ask what should change when the equation is manipulated, then let the visualization test that expectation.
Foundation Lab
Probing, Linear Classifier Probes & Activation Analysis
Probing is one of the main tools to understand what GPT-like models know and where that knowledge lives
#12ProbingRepresentations
key equation
\hat y = W h_\ell(x) + bSelected Foundation Object
Keep the equation fixed; move through the evidence.
Concept 12 of 100ProbingRepresentations / Phase 5: Representation & interpretability
Current question
Clear mental model of what probes measure (information content vs ease of extraction)
\hat y = W h_\ell(x) + bUse the runnable panel, the key equation, and canonical papers as separate forms of evidence for the same object.
The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.
Use prerequisites, dependents, and semantic links to repair the next gap without leaving the object behind.
Why It Matters for Modern Models
- Probing is one of the main tools to understand what GPT-like models know and where that knowledge lives
- Used heavily for safety (probing for dangerous capabilities), robustness, and fairness analyses
What Tutorials Skip
What is still poorly explained in textbooks and papers:
- Clear mental model of what probes measure (information content vs ease of extraction)
- Visual, layer-by-layer maps of information flow in large LMs
Interactive Visualization
Core Math (Optional Deep Dive)
If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.
Key Equation
Given layer representation , train a frozen probe:
(or a softmax over ) on a supervised task (POS tags, parse trees, etc.). The accuracy estimates how linearly separable that information is at layer .
BERT layers roughly follow the classical NLP pipeline (POS → syntax → semantics → coreference).