Capability elicitation estimates the best score a model can reach on a task by varying the prompt, the scaffolding around the model and any fine-tuning. Greenblatt et al. (2024) deliberately locked capabilities behind a password and found that a few high-quality demonstrations were often enough to unlock them by fine-tuning, and that reinforcement learning often worked when only evaluations were available.
Concept 93 of 100Scaling & AlignmentStudy phase 12
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Capability elicitation estimates the best score a model can reach on a task by varying the prompt, the scaffolding around the model and any fine-tuning. Greenblatt et al. (2024) deliberately locked capabilities behind a password and found that a few high-quality demonstrations were often enough to unlock them by fine-tuning, and that reinforcement learning often worked when only evaluations were available.
Burns et al. (2023) found directions in a model's activations that satisfy logical consistency, such as a statement and its negation having opposite truth values, without any labels; this beat zero-shot accuracy by 4% on average and stayed accurate when the model was prompted to give wrong answers.
Whether such methods find knowledge is contested: Farquhar et al. (2023) showed that unsupervised methods of this kind tend to pick up whatever feature of the activations is most prominent, which need not be the model's knowledge.
Questions to Think About
A model may represent a fact that it does not state; what would it mean for a reporter to read what the model knows rather than what a person would say (Christiano et al., 2021)?
Why can a consistency-based probe find a feature that is not truth at all, and how could one tell the difference (Farquhar et al., 2023)?
Core Math
Key Equation
gψ(h(x))≈z
Two related questions go under the name elicitation.
Capability elicitation asks what a model can do when it is used well. A measured score depends on the prompt, the scaffolding and any fine-tuning, so the quantity of interest is closer to the best score over a set of elicitation methods P,
Cap(M)=p∈PmaxE[score(M,p)],
and any single evaluation gives only a lower bound on it. Evaluations that test for dangerous capabilities need the top of this range, not the model's default behaviour.
Eliciting latent knowledge (ELK) asks how to get a model to report what it internally represents, even when its outputs would say something else (Christiano et al., 2021). The report's example is a model that predicts camera images and "knows" that a camera has been tampered with, although the images look normal. One approach trains a reporter gψ on the model's hidden activations h(x) to recover the truth z:
gψ(h(x))≈z
The difficulty is that a reporter trained on human labels may learn to predict what a person would believe instead of what the model knows.
Key Papers
ARC's First Technical Report: Eliciting Latent Knowledge