Foundation Lab

Preference-Based Alignment: RLHF, Reward Modeling, Constitutional AI

GPT-4, Claude-3, Gemini rely on RLHF-style procedures to be helpful, honest, harmless

Concept 15 of 100Scaling & AlignmentPhase 7

Open New Notebook All Foundations

#15RLHFScaling & Alignment

key equation\max_\theta \mathbb E_{\pi_\theta}[r_\phi(x,y)] - \beta\, \mathrm{KL}(\pi_\theta\,\|\,\pi_0)

Selected Foundation Object

Keep the equation fixed; move through the evidence.

Concept 15 of 100RLHFScaling & Alignment / Phase 7: Alignment & RLHF

Current question

Conceptual explanation of RLHF as a KL-regularized Bayesian update on behavior

\max_\theta \mathbb E_{\pi_\theta}[r_\phi(x,y)] - \beta\, \mathrm{KL}(\pi_\theta\,\|\,\pi_0)

PredictionCommit before the demo.

Ask what should change when the equation is manipulated, then let the visualization test that expectation.

EvidenceCompare local witness and source.

Use the runnable panel, the key equation, and canonical papers as separate forms of evidence for the same object.

InvariantName what survives notation changes.

The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.

Next moveContinue in the newer notebook.

This atlas page keeps the working demo; the domain notebook carries the fuller Intuition -> Math -> Code -> Demo sequence.

Open demo Inspect equation Open newer notebook Next foundation

Why It Matters for Modern Models

GPT-4, Claude-3, Gemini rely on RLHF-style procedures to be helpful, honest, harmless
Constitutional AI ideas are key to Anthropic's Claude models

What Tutorials Skip

What is still poorly explained in textbooks and papers:

Conceptual explanation of RLHF as a KL-regularized Bayesian update on behavior
How over-optimization of learned reward leads to reward hacking and distribution shift
Interactive visualizations of policy distributions before/after RLHF

Interactive Visualization

Core Math (Optional Deep Dive)

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation

\max_\theta \mathbb E_{\pi_\theta}[r_\phi(x,y)] - \beta\, \mathrm{KL}(\pi_\theta\,\|\,\pi_0)

Reward modeling from preferences: Given human comparisons between outputs $y_a, y_b$ , learn reward model via Bradley–Terry:

P(y_a \succ y_b \mid x) = \frac{\exp(r_\phi(x,y_a))}{\exp(r_\phi(x,y_a)) + \exp(r_\phi(x,y_b))}

RLHF objective: Fine-tune policy $\pi_\theta(y\mid x)$ to maximize reward while staying close to reference model $\pi_0$ :

\max_\theta \mathbb E_{x,y\sim \pi_\theta}[r_\phi(x,y)] - \beta\, \mathrm{KL}(\pi_\theta(\cdot\mid x)\,\|\,\pi_0(\cdot\mid x))

Constitutional AI: "labeler" is another model guided by a constitution (natural-language principles).

Canonical Papers

Deep Reinforcement Learning from Human Preferences

Christiano et al.2017NeurIPS

Read paper →

Training Language Models to Follow Instructions with Human Feedback

Ouyang et al.2022NeurIPS (InstructGPT)

Read paper →

Constitutional AI: Harmlessness from AI Feedback

Bai et al.2022Anthropic

Read paper →

Connections

Prerequisites

ℒML/CE/KL ↗Scaling

Enables

🎯DPO 👍KTO ⚠️Reward Hacking 🎯PPO 📜Constitutional 🧠Test-Time 🏭Synth Data 📋PRMs 🤖RLAIF ⚔️Debate 🔄IDA 💪Weak→Strong 😴Sleepers 🥪Sandwich 🔄Self-Improve

Next Moves

Choose the next question to carry this object forward.

Semantic Connections

Break / Breaks When

🎯Reward hacking→DPO ⚠️Proxy gaming→Reward Hacking 🎲Tail sampling breaks alignment→Decoding

Fix / Invented to Fix

👍Preference pairs→KTO

Dual / Mathematical Dual

🎯RL ↔ Classification→DPO

Reuse / Same Technique

ℒCross-entropy→ML/CE/KL ⚡KL anchoring→Efficiency

Map / Analogy

⚔Learned critic→GANs