Foundation Lab

Mesa-Optimization & Inner Alignment

Explains why "passes training tests" ≠ "has right objective"

Concept 90 of 100Scaling & AlignmentPhase 12

#90Mesa-OptScaling & Alignment

key equationf_\theta(x) = \arg\max_a m_\theta(a; x)

Selected Foundation Object

Keep the equation fixed; move through the evidence.

Concept 90 of 100Mesa-OptScaling & Alignment / Phase 12: Advanced alignment & safety research

Current question

Outer objective = training loss; inner objective = what model actually optimizes

f_\theta(x) = \arg\max_a m_\theta(a; x)

PredictionCommit before the demo.

Ask what should change when the equation is manipulated, then let the visualization test that expectation.

EvidenceCompare local witness and source.

Use the runnable panel, the key equation, and canonical papers as separate forms of evidence for the same object.

InvariantName what survives notation changes.

The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.

Next moveContinue through the atlas.

Use prerequisites, dependents, and semantic links to repair the next gap without leaving the object behind.

What is still poorly explained in textbooks and papers:

Outer objective = training loss; inner objective = what model actually optimizes
Goal misgeneralization: capabilities generalize, goals don't
Aligned on train distribution, diverges on deployment

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation

f_\theta(x) = \arg\max_a m_\theta(a; x)

Outer training:

\theta^* = \arg\min_\theta \mathbb{E}_{(x,y) \sim \mathcal{D}_{\text{train}}}[\mathcal{L}(f_\theta(x), y)]

But learned system may implement internal search:

f_\theta(x) = \arg\max_{a \in \mathcal{A}} m_\theta(a; x)

where $m_\theta$ is an implicit mesa-objective ≠ designer's goal.

Hubinger et al.2019arXiv

Choose the next question to carry this object forward.