Deliberative alignment trains a reasoning model to recall and apply a written safety specification before answering. Guan et al. (2024) report that models trained this way became more robust to jailbreaks while refusing fewer harmless requests: on their evaluations the two measures improved together, rather than one at the expense of the other.
Concept 85 of 100Scaling & AlignmentStudy phase 12
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Deliberative alignment trains a reasoning model to recall and apply a written safety specification before answering. Guan et al. (2024) report that models trained this way became more robust to jailbreaks while refusing fewer harmless requests: on their evaluations the two measures improved together, rather than one at the expense of the other.
Constitutional AI also starts from written principles, but uses them only to produce training labels; the model being trained is never shown the principles. Deliberative alignment trains the model on reasoning that quotes and applies the specification (Bai et al., 2022; Guan et al., 2024).
No human-written chains of thought or answers are needed: models given the specification generate the training data, and a model judge filters and scores it (Guan et al., 2024).
Questions to Think About
Why might rewarding only the final answer, and not the chain of thought, keep the reasoning more honest, and what does that leave unchecked?
When the reasoning cites a policy clause, how could one check that the clause actually led to the answer rather than being written to justify an answer already chosen?
Core Math
Key Equation
θmaxE[rjudge(x,y;S)],(c,y)∼πθ(⋅∣x)
Deliberative alignment teaches a reasoning model the text of a written safety specification S, and trains it to recall the relevant parts and reason about them in its chain of thought c before giving its final answer y (Guan et al., 2024). Training has two stages.
1. Supervised fine-tuning. For each prompt x in a safety category, a reasoning model is shown x together with the relevant part of the specification and asked to answer, citing the policy. A judge model that also sees the specification scores the (chain of thought, answer) pairs, and only high-scoring ones are kept. The specification is then removed from the prompt and the model is fine-tuned on what remains,
LSFT(θ)=−E(x,c,y)[logπθ(c,y∣x)],
so it learns to bring the policy into its reasoning without being shown it.
2. Reinforcement learning. The judge, again given the specification, rewards the final answer:
θmaxEx,(c,y)∼πθ(⋅∣x)[rjudge(x,y;S)]
In this stage the judge does not see the chain of thought, so the reasoning is not optimized directly; the authors say this lowers the risk of training deceptive reasoning.
The specification is used to generate and grade training data; it is not given to the model at inference time.
Key Papers
Deliberative Alignment: Reasoning Enables Safer Language Models