Reinforcement learning from AI feedback (RLAIF) uses a language model's preferences between pairs of responses in place of human preference labels. On summarization, helpful dialogue and harmless dialogue, it performed comparably to RLHF in human evaluations (Lee et al., 2024).
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Reinforcement learning from AI feedback (RLAIF) uses a language model's preferences between pairs of responses in place of human preference labels. On summarization, helpful dialogue and harmless dialogue, it performed comparably to RLHF in human evaluations (Lee et al., 2024).
Lee et al. (2024) estimate AI preference labels to cost over 10 times less than human ones. Their best labelling prompt agreed with human preferences on 78% of summarization comparisons, and RLAIF improved on a supervised fine-tuned baseline even when the labelling model was the same checkpoint as the starting policy.
Constitutional AI trained a model to be less harmful using AI preference labels, with no human labels of harmful outputs; the human input was a short list of written principles (Bai et al., 2022).
Questions to Think About
Why does an AI labeller's preference depend on which response it sees first, and why does averaging over both orders reduce the problem?
Which judgments can safely be delegated to a model given a clear rubric, and which still need people?
Core Math
Key Equation
P(y1≻y2∣x)=LLM(x,y1,y2,rubric)
Reinforcement learning from AI feedback (RLAIF) keeps the RLHF pipeline but replaces the person who compares two responses with a language model.
AI preference labels. For a prompt x and two responses y1,y2, an off-the-shelf model is shown the pair, often with a rubric or a list of principles, and its output is read as a preference probability:
P(y1≻y2∣x)=LLM(x,y1,y2,rubric)
Lee et al. (2024) ask twice, with the two responses in both orders, and average the results, because the labelling model favours responses in particular positions.
Reward model. Fit rϕ to these labels with the usual pairwise loss −logσ(rϕ(x,yw)−rϕ(x,yl)), where yw is the preferred response and yl the other.
Policy optimization. Maximize rϕ with a KL penalty that keeps the policy close to where it started, as in RLHF.
In a variant called direct RLAIF, the labelling model scores responses during RL and no separate reward model is trained (Lee et al., 2024).
Key Papers
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback