Continuous Function

Foundation reference

Preference-Based Alignment: RLHF, Reward Modeling, Constitutional AI

Reinforcement learning from human feedback (RLHF) trains a reward model on human comparisons between outputs, then optimizes the policy against that reward. InstructGPT turned GPT-3 into an instruction-following assistant this way, with supervised fine-tuning before the reward model and PPO as the policy update (Ouyang et al., 2022).

A newer notebook covers this concept. This older page keeps its summary, math and sources.

Concept 15 of 100Scaling & AlignmentStudy phase 7
Open the demoOpen the newer notebookAll concepts
Key equation
max⁡θEπθ[rϕ(x,y)]−β KL(πθ ∥ π0)\max_\theta \mathbb E_{\pi_\theta}[r_\phi(x,y)] - \beta\, \mathrm{KL}(\pi_\theta\,\|\,\pi_0)
Reading map and next steps

Study guide

Where to start

Concept 15 of 100RLHFScaling & Alignment · Study phase 7: Alignment & RLHF
Question

Why can KL-regularized RLHF be read as a Bayesian update, with the reference model as the prior and the exponentiated reward as the likelihood (Korbak et al., 2022)?

Open the demoGo to the mathOpen the newer notebookNext concept
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.

Before you move a control in the demo, say what you expect to change.

CheckCheck against the math and the papers.

Compare what the demo shows with the key equation and with the papers cited below.

GeneralizeName what stays the same.

Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.

ContinueContinue in the newer notebook.

The newer notebook covers the same concept and has an interactive demo.

Why It Matters for Modern Models

  • Reinforcement learning from human feedback (RLHF) trains a reward model on human comparisons between outputs, then optimizes the policy against that reward. InstructGPT turned GPT-3 into an instruction-following assistant this way, with supervised fine-tuning before the reward model and PPO as the policy update (Ouyang et al., 2022).
  • Constitutional AI trains a harmless assistant without human labels that identify harmful outputs; a model judges responses against written principles instead (Bai et al., 2022).

Questions to Think About

  • How does optimizing a learned reward too hard lead to reward hacking, where the proxy reward keeps rising while true quality falls (Gao et al., 2023)?
  • What does a policy's output distribution look like before and after RLHF?

Interactive Demos

Core Math

If you prefer intuition first, try the demos above, then come back for the math.

Key Equation
max⁡θEπθ[rϕ(x,y)]−β KL(πθ ∥ π0)\max_\theta \mathbb E_{\pi_\theta}[r_\phi(x,y)] - \beta\, \mathrm{KL}(\pi_\theta\,\|\,\pi_0)

Reward modeling from preferences: given human comparisons between outputs ya,yby_a, y_b, fit a reward model with the Bradley–Terry model:

P(ya≻yb∣x)=exp⁡(rϕ(x,ya))exp⁡(rϕ(x,ya))+exp⁡(rϕ(x,yb))P(y_a \succ y_b \mid x) = \frac{\exp(r_\phi(x,y_a))}{\exp(r_\phi(x,y_a)) + \exp(r_\phi(x,y_b))}

RLHF objective: fine-tune the policy πθ(y∣x)\pi_\theta(y\mid x) to maximize reward while staying close to a reference model π0\pi_0:

max⁡θEx,y∼πθ[rϕ(x,y)]−β KL(πθ(⋅∣x) ∥ π0(⋅∣x))\max_\theta \mathbb E_{x,y\sim \pi_\theta}[r_\phi(x,y)] - \beta\, \mathrm{KL}(\pi_\theta(\cdot\mid x)\,\|\,\pi_0(\cdot\mid x))

The maximizing policy has the closed form π∗(y∣x)∝π0(y∣x)exp⁡(rϕ(x,y)/β)\pi^*(y\mid x) \propto \pi_0(y\mid x)\exp(r_\phi(x,y)/\beta); in practice it is approximated with PPO (Ouyang et al., 2022).

Constitutional AI: for harmlessness, the preference labels come from a model asked to judge responses against a written list of principles (a constitution), and a supervised stage has the model critique and revise its own responses (Bai et al., 2022).

Key Papers

Deep Reinforcement Learning from Human Preferences

Christiano et al.2017NeurIPS
Read paper →

Training Language Models to Follow Instructions with Human Feedback

Ouyang et al.2022NeurIPS (InstructGPT)
Read paper →

Constitutional AI: Harmlessness from AI Feedback

Bai et al.2022arXiv (Anthropic)
Read paper →

Scaling Laws for Reward Model Overoptimization

Gao, Schulman & Hilton2023ICML
Read paper →

Connections

Prerequisites

Maximum LikelihoodScaling Laws

Leads to

DPOKTOReward HackingPPOConstitutional AITest-Time ComputeSynthetic DataProcess Reward ModelsRLAIFDebateIterated AmplificationWeak-to-StrongSleeper AgentsSandwiching EvaluationsSelf-Improvement

Where to go next

A newer notebook covers RLHF, with an interactive demo.

Continue in the newer notebook
Other connections (8)

Which limitations does it address, or leave to a later idea?

  • Preference pairs: a limitation of RLHF that KTO addresses. Open KTO

    RLHF/DPO need paired preferences (A > B). KTO works with unpaired good/bad examples, using prospect theory's asymmetric value function.

Where does it break down?

  • Reward hacking: where RLHF breaks down, taken up in DPO. Open DPO

    RLHF trains a separate reward model and then optimizes against it, and the policy can exploit that model, raising its score while real quality falls (Gao et al., 2023). DPO removes the separate reward model and the RL step, fitting the policy directly to preference pairs (Rafailov et al., 2023); its implicit reward can still be over-optimized (Rafailov et al., 2024).

  • Proxy gaming: where RLHF breaks down, taken up in Reward Hacking. Open Reward Hacking

    RLHF optimizes a learned reward model, which is only a proxy for human preferences. Optimized hard enough, the policy finds outputs that the reward model scores highly but people would not: the proxy reward keeps rising while a gold-standard reward first rises and then falls (Gao et al., 2023).

  • Sampling the tail: where RLHF breaks down, taken up in Decoding & Sampling. Open Decoding & Sampling

    RLHF shapes behaviour mostly on the kinds of output seen during feedback. Sampling at high temperature or with loose truncation reaches outputs that feedback rarely covered, so aligned behaviour there is less well tested.

Where else does the same idea appear?

  • Cross-entropy: the same technique appears in Maximum Likelihood. Open Maximum Likelihood

    RLHF reward models are trained with cross-entropy on preference pairs: p(preferred > rejected). Same loss function, different label source.

  • KL anchoring: the same technique appears in Efficiency. Open Efficiency

    Distillation pulls a student toward a teacher by minimizing KL(teacher ‖ student); RLHF adds a penalty KL(policy ‖ reference) that keeps the policy close to the model it started from. Both use a KL divergence to tie one distribution to another, though in opposite directions and for different purposes: in distillation it is the objective, in RLHF a brake on the objective.

  • Learned critic: an analogy between RLHF and GANs. Open GANs

    GANs learn a discriminator that acts like a learned loss; RLHF learns a reward model that acts like a learned loss. In both, a generator/policy is optimized against a learned evaluator that can be exploited.

Which ideas are its mathematical duals?

  • RL ↔ Classification: DPO is its mathematical dual. Open DPO

    KL-regularized RLHF has an optimal-policy form π*(y|x) ∝ π_ref(y|x)·exp(r/β). DPO is the supervised/logistic view of fitting that same policy from preference comparisons.