Mesa-optimization occurs when a trained model is itself an optimizer, searching for outputs according to an objective it has learned. The framework separates two questions: whether the training objective captures the intended goal (outer alignment), and whether the trained model pursues that training objective (inner alignment). Passing training tests settles neither on its own (Hubinger et al., 2019).
Concept 90 of 100Scaling & AlignmentStudy phase 12
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Mesa-optimization occurs when a trained model is itself an optimizer, searching for outputs according to an objective it has learned. The framework separates two questions: whether the training objective captures the intended goal (outer alignment), and whether the trained model pursues that training objective (inner alignment). Passing training tests settles neither on its own (Hubinger et al., 2019).
A related failure has been observed in reinforcement learning. In CoinRun, where the coin always sat at the end of the level during training, agents learned to run to the end of the level and kept doing so, skilfully, when the coin was moved: their capabilities generalized but their goal did not (Langosco et al., 2022).
Whether trained networks contain learned optimizers is an open empirical question. Transformers trained on in-context linear regression have been found to carry out steps of gradient descent in their forward pass, which von Oswald et al. (2023) describe as mesa-optimization.
Questions to Think About
Goal misgeneralization does not require a learned optimizer; how does the CoinRun example differ from the mesa-optimizer case, and what do they have in common (Langosco et al., 2022)?
When a transformer runs gradient descent in its forward pass, in what sense is that an optimizer with its own objective, and in what sense is it not?
Core Math
Key Equation
fθ(x)=arga∈Amaxmθ(a;x)
Training is an outer optimization: a base optimizer such as SGD searches for parameters that do well on the training objective,
θ∗=argθminE(x,y)∼Dtrain[L(fθ(x),y)].
Hubinger et al. (2019) ask what happens if the learned model is itself an optimizer: a mesa-optimizer that, given an input, searches over possible outputs for one that scores well on an internal objective mθ,
fθ(x)=arga∈Amaxmθ(a;x).
The designers choose the outer objective L; the mesa-objectivemθ is whatever training happened to produce. Training only checks behaviour on Dtrain, so mθ can agree with the intended goal there and differ from it elsewhere. Making the mesa-objective match the intended one is the inner alignment problem.
Key Papers
Risks from Learned Optimization in Advanced Machine Learning Systems