Constitutional AI replaces human labels of harmful outputs with labels from a model guided by written principles; Bai et al. (2022) report an assistant that is harmless but not evasive, engaging with harmful requests by explaining its objections to them.
Concept 69 of 100Constitutional AIScaling & Alignment · Study phase 7: Alignment & RLHF
Question
What does a constitution contain, and how does drawing a different principle at random for each critique or comparison change the result compared with using one principle throughout?
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Constitutional AI replaces human labels of harmful outputs with labels from a model guided by written principles; Bai et al. (2022) report an assistant that is harmless but not evasive, engaging with harmful requests by explaining its objections to them.
Anthropic has said that its Claude models are trained with Constitutional AI, using principles updated from those in the paper (Anthropic, 2023).
The principles are written in plain language, so the intended behavior can be read and edited directly; Bai et al. (2022) argue that this allows more precise control of behavior with far fewer human labels.
AI feedback has since been compared directly with human feedback: on summarization and on helpful and harmless dialogue, Lee et al. (2024) found that RLAIF performed comparably to RLHF.
Questions to Think About
Which judgements does Constitutional AI still take from people, and which does it hand to the model?
Bai et al. (2022) found that each further revision made responses more harmless by their preference model's measure but less helpful; why might repeated self-critique trade one for the other?
Core Math
Key Equation
c=LM(prompt,r0,principle),r1=LM(prompt,r0,c)
Constitutional AI trains a harmless assistant without human labels that mark outputs as harmful. The human input is a short list of written principles, the constitution; the Constitutional AI paper used 16 principles about harmlessness (Bai et al., 2022). Training has two phases.
1. Supervised phase: critique and revision. The model answers a prompt, critiques its own answer according to a principle drawn at random from the constitution, and rewrites the answer in the light of the critique:
c=LM(prompt,r0,principle),r1=LM(prompt,r0,c)
The step can be repeated with new principles, and the pretrained model is then fine-tuned on the final revisions.
2. Reinforcement learning from AI feedback (RLAIF). The fine-tuned model produces pairs of answers, and a model judges, according to a randomly drawn principle, which answer is better. A preference model rθ is trained on these AI labels with the usual pairwise loss
LPM=−logσ(rθ(x,ychosen)−rθ(x,yrejected))
and the policy is trained with reinforcement learning against rθ. In the paper, comparisons for helpfulness still came from human labels; only those for harmlessness came from the model.