Sparse Autoencoders at Scale: Feature Dictionaries for Mechanistic Interpretability
A sparse autoencoder reconstructs an activation vector using a few learned feature directions. These features can be more interpretable individually than neurons, which often respond to several unrelated concepts (Bricken et al., 2023).
A newer notebook covers this concept. This older page keeps its summary, math and sources.
Concept 27 of 100Sparse AutoencodersRepresentations · Study phase 5: Representation & interpretability
Question
Why is it useful to think of an SAE as learning a dictionary of parts for the residual stream, with each decoder column a candidate feature and the sparse code saying which parts are present?
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you move a control in the demo, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the demo shows with the key equation and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueContinue in the newer notebook.
The newer notebook covers the same concept and has an interactive demo.
Why It Matters for Modern Models
A sparse autoencoder reconstructs an activation vector using a few learned feature directions. These features can be more interpretable individually than neurons, which often respond to several unrelated concepts (Bricken et al., 2023).
The method has been scaled to large production models: Gao et al. (2025) trained a 16-million-latent autoencoder on GPT-4 activations, and Templeton et al. (2024) extracted interpretable features from Claude 3 Sonnet.
It turns the idea of superposition into a practical procedure for recovering features from real models.
Features found this way can be the units of circuit analysis, giving causal graphs over interpretable features rather than over attention heads and neurons (Marks et al., 2025).
Once a feature can be named, it can be measured, audited and intervened on, which links interpretability to alignment work such as activation steering.
Questions to Think About
A dense code can represent anything but names nothing; how does sparsity force the same direction to be reused across similar contexts?
In a TopK SAE, k sets how many features may explain one activation; what is lost when k is too small, and what when it is too large?
What causes the L1 penalty's shrinkage bias and dead latents, and how do gated and TopK SAEs address them?
What evidence shows that an SAE feature is used by the model, rather than only correlated with a concept, and how do ablation and steering provide it?
Interactive Demo
Core Math
If you prefer intuition first, try the demo above, then come back for the math.
Key Equation
z=TopK(Wenc(x−bpre),k)
A sparse autoencoder (SAE) learns a dictionary of directions in a model's activation space so that each activation is explained by a few of them. The motivation is superposition: a network can represent more features than it has dimensions by storing them as overlapping directions, so individual neurons often respond to several unrelated things.
Let x∈Rd be an activation vector, for example the residual stream at one layer.
Encode to sparse latents, decode to a reconstruction:
z=ReLU(Wenc(x−bpre)+benc),x^=Wdecz+bpre
Each column of Wdec is a unit-norm candidate feature direction, and the sparse code z says which features are present and how strongly.
Standard objective (reconstruction plus an L1 sparsity penalty):
L=∥x−x^∥22+λ∥z∥1
Sparsity is the assumption that makes latents more likely to be interpretable: if only a few latents can be active at once, each tends to specialize. The L1 penalty also shrinks active values toward zero, a bias that gated SAEs were designed to remove (Rajamanoharan et al., 2024).
TopK variant (set the sparsity directly):
z=TopK(Wenc(x−bpre),k),L=∥x−x^∥22
Only the k largest pre-activations are kept, so k fixes the number of active latents per token instead of being tuned indirectly through λ. Gao et al. (2025) report that this improves the trade-off between reconstruction error and sparsity and leaves few dead latents even at large scale.
Which limitations does it address, or leave to a later idea?
Disentangle features: a limitation of Sparse Autoencoders that Superposition addresses. Open Superposition
Sparse autoencoders do not remove superposition from the model; they read features out of it. They learn more directions than the layer has neurons and explain each activation with a few of them, so the learned features are more often monosemantic than the neurons are (Bricken et al., 2023).
Feature dictionaries: a limitation of Superposition that Sparse Autoencoders addresses. Open Superposition
Superposition is one explanation for why neurons are polysemantic and hard to interpret (Elhage et al., 2022). Sparse autoencoders were developed to learn an overcomplete dictionary that recovers sparse, more interpretable features from dense activations (Bricken et al., 2023).
Targeted intervention: a limitation of Activation Steering that Sparse Autoencoders addresses. Open Activation Steering
SAE features make steering more targeted. Instead of adding a direction found by contrasting two sets of examples, one can add or remove a specific feature that the SAE has identified and that has a human-readable description (Templeton et al., 2024).
Where does it break down?
Feature entanglement: where Activation Steering breaks down, taken up on this page. Open Activation Steering
SAE features are not guaranteed to be perfectly monosemantic or causally isolated. Steering along imperfect features can cause unintended side effects on behaviour, or push the hidden states into regions the model never saw in training.
Where else does the same idea appear?
Top-k gating: the same technique appears in MoE. Open MoE
k-sparse SAEs enforce TopK activations; MoE routing enforces TopK experts per token. Both implement sparse selection as "keep top-k, drop the rest, then compute through a smaller active set."
Dictionary learning: the same technique appears in Embeddings. Open Embeddings
SAEs apply dictionary learning to a network's activations: each activation is explained as a sparse combination of learned directions, which are often easier to interpret than individual neurons. The dictionary has more directions than the activations have dimensions, so it is not a basis.
Codebook sparsity: an analogy between Sparse Autoencoders and Tokenization. Open Tokenization
Tokenization represents text by selecting items from a learned codebook (vocab). SAEs represent activations by selecting items from a learned codebook (features)—both are "dictionary + sparse code" stories.