15Scaling & Alignment

⚖Preference-Based Alignment: RLHF, Reward Modeling, Constitutional AI

Canonical Papers

Christiano et al.2017NeurIPS

Ouyang et al.2022NeurIPS (InstructGPT)

Bai et al.2022Anthropic

Reward modeling from preferences: Given human comparisons between outputs $y_a, y_b$ , learn reward model via Bradley–Terry:

P(y_a \succ y_b \mid x) = \frac{\exp(r_\phi(x,y_a))}{\exp(r_\phi(x,y_a)) + \exp(r_\phi(x,y_b))}

RLHF objective: Fine-tune policy $\pi_\theta(y\mid x)$ to maximize reward while staying close to reference model $\pi_0$ :

\max_\theta \mathbb E_{x,y\sim \pi_\theta}[r_\phi(x,y)] - \beta\, \mathrm{KL}(\pi_\theta(\cdot\mid x)\,\|\,\pi_0(\cdot\mid x))

Constitutional AI: "labeler" is another model guided by a constitution (natural-language principles).

Key Equation

\max_\theta \mathbb E_{\pi_\theta}[r_\phi(x,y)] - \beta\, \mathrm{KL}(\pi_\theta\,\|\,\pi_0)

GPT-4, Claude-3, Gemini rely on RLHF-style procedures to be helpful, honest, harmless
Constitutional AI ideas are key to Anthropic's Claude models

What is still poorly explained in textbooks and papers:

Conceptual explanation of RLHF as a KL-regularized Bayesian update on behavior
How over-optimization of learned reward leads to reward hacking and distribution shift
Interactive visualizations of policy distributions before/after RLHF

Explore this concept from different angles — like a mathematician would.

⚠️ Breaks When

🔧 Invented to Fix

↔️ Mathematical Dual

🔄 Same Technique

≈ Analogy