Mechanistic Interpretability

What Do Networks Compute?

Mechanistic interpretability tries to reverse-engineer what a neural network computes into algorithms a person can understand (Olah et al., 2020). It reads the network's activations, and intervenes on them, to test such an explanation. The figures on this page are illustrative, not recordings from a trained model.

The goal: not just that a network works, but how it works, from the features it represents to the circuits that connect them.

Reading Activations

A first step is to ask what individual neurons respond to. In vision models, some neurons respond to recognizable features such as curves or dog heads; others respond to mixtures of unrelated inputs (Olah et al., 2020).

The figure shows what such a recording looks like, with time steps across and neurons down. The Sparsity control sets the chance that a neuron is nearly silent at a given step.

Superposition

A key difficulty: a network may represent more features than it has neurons. In toy models, Elhage et al. (2022) showed that when features are sparse, a network stores them in superposition, as overlapping directions that interfere slightly with each other.

One consequence is polysemantic neurons, which respond to several unrelated features. Looking at neurons one at a time is then not enough.

Sparse Autoencoders

Sparse autoencoders (SAEs) try to undo superposition. They learn a dictionary of feature directions, more numerous than the neurons, and rebuild each activation vector x from a few of them (Bricken et al., 2023):

The columns of W_dec are the dictionary features, constrained to unit norm so decoder growth cannot cancel the sparsity penalty. The L1 penalty pushes each input to be explained by a few features, in the hope that each feature then stands for one interpretable concept.

The figure shows a simpler, supervised relative: a linear probe, a single direction in activation space chosen to predict a known label (here, sentiment). An SAE instead learns thousands of directions without labels.

Feature Interpretation

Once we have features, we can study them:

• → Find examples that maximally activate each feature
• → Ablate features to see their causal effect
• → Track how features compose across layers

Whether this can add up to a full account of a model is an open question. An SAE reconstructs activations only approximately, and the features it finds depend on the size of its dictionary: in larger dictionaries, features split into finer ones (Bricken et al., 2023).

Circuits

Features don't act in isolation; they connect into circuits. A circuit is a subgraph of the network's computation that implements a specific behaviour (Olah et al., 2020).

Circuits identified in language models include:

• • Induction heads, which find an earlier occurrence of the current token and copy what followed it; Olsson et al. (2022) link them to in-context learning
• • Indirect object identification in GPT-2 small (Wang et al., 2023)
• • Greater-than comparison of years in GPT-2 small (Hanna et al., 2023)

A circuit gives a mechanistic account of a behaviour, which can also help explain when the behaviour fails.