Iterated amplification trains a model to imitate a person who breaks a task into sub-questions and combines the current model's answers to them. It is a proposal for supervising models on tasks too hard for people to evaluate directly: no one has to judge the whole task, only to split questions and combine answers that are each easy to check (Christiano et al., 2018).
Concept 87 of 100Scaling & AlignmentStudy phase 12
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Iterated amplification trains a model to imitate a person who breaks a task into sub-questions and combines the current model's answers to them. It is a proposal for supervising models on tasks too hard for people to evaluate directly: no one has to judge the whole task, only to split questions and combine answers that are each easy to check (Christiano et al., 2018).
Christiano et al. (2018) tested it on algorithmic tasks, where it learned complex behavior without any external reward function.
Recursive decomposition has been used at scale: Wu et al. (2021) summarized entire novels with a model that, trained with human feedback, summarizes short sections and then recursively summarizes the summaries.
Questions to Think About
Each distillation step imitates the amplified procedure imperfectly; why might small errors build up over many rounds?
How does iterated amplification differ from debate, where two models argue and a person judges, as a way to supervise models on hard tasks?
Core Math
Key Equation
A′=argπminExKL(Amp(H,A)(⋅∣x)∥π(⋅∣x))
Iterated amplification builds a training signal for hard tasks out of answers to easier sub-questions (Christiano et al., 2018).
Amplify. A human H answers a question x by breaking it into sub-questions and asking the current model A to answer each one. Call this combined procedure Amp(H,A): it is slower than A alone, but more capable.
Distill. Train a new model to imitate the amplified procedure:
A′=argπminEx[KL(Amp(H,A)(⋅∣x)∥π(⋅∣x))]
Iterate. Set A←A′ and repeat.
Each round, the human working with a better model can handle harder questions, so the training signal keeps pace with the model without anyone having to solve the full task. The scheme depends on hard questions breaking down into easier ones that a person can coordinate.
Key Papers
Supervising strong learners by amplifying weak experts