Continuous Function

Foundation reference

Direct Preference Optimization: RL-Free Alignment from Human Preferences

DPO trains a model on preference pairs with a single classification-style loss, without a separate reward model or a reinforcement-learning loop (Rafailov et al., 2023).

A newer notebook covers this concept. This older page keeps its summary, math and sources.

Concept 24 of 100Scaling & AlignmentStudy phase 7
Open the demoOpen the newer notebookAll concepts
Key equation
LDPO=−E[log⁡σ ⁣(β[log⁡πθ(yw∣x)πref(yw∣x)−log⁡πθ(yℓ∣x)πref(yℓ∣x)])]\mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[\log \sigma\!\left(\beta \left[\log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log\frac{\pi_\theta(y_\ell|x)}{\pi_{\text{ref}}(y_\ell|x)}\right]\right)\right]
Reading map and next steps

Study guide

Where to start

Concept 24 of 100DPOScaling & Alignment · Study phase 7: Alignment & RLHF
Question

Why is DPO better described as moving probability mass between responses than as learning a scalar reward?

Open the demoGo to the mathOpen the newer notebookNext concept
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.

Before you move a control in the demo, say what you expect to change.

CheckCheck against the math and the papers.

Compare what the demo shows with the key equation and with the papers cited below.

GeneralizeName what stays the same.

Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.

ContinueContinue in the newer notebook.

The newer notebook covers the same concept and has an interactive demo.

Why It Matters for Modern Models

  • DPO trains a model on preference pairs with a single classification-style loss, without a separate reward model or a reinforcement-learning loop (Rafailov et al., 2023).
  • Several open models were post-trained with DPO, including Zephyr-7B (Tunstall et al., 2023) and Llama 3 (Grattafiori et al., 2024).
  • Variants change the loss: SimPO drops the reference model and uses length-normalized log-probabilities (Meng et al., 2024), and DPO-Positive adds a term that keeps the likelihood of the preferred response from falling (Pal et al., 2024).
  • DPO makes explicit that preference training is a KL-regularized reshaping of the reference model's distribution, whether or not reinforcement learning is used.

Questions to Think About

  • What role does the reference model play, and why does the KL term act like a trust region that keeps the policy where the preference data are informative?
  • How can DPO widen the gap between preferred and rejected responses while lowering the likelihood of the preferred one (Pal et al., 2024)?
  • What can offline preference optimization learn about situations that its preference data never cover?
  • In what sense is DPO logistic regression on log-probability ratios, and what does it keep and lose relative to RLHF with an explicit reward model?

Interactive Demo

Core Math

If you prefer intuition first, try the demo above, then come back for the math.

Key Equation
LDPO=−E[log⁡σ ⁣(β[log⁡πθ(yw∣x)πref(yw∣x)−log⁡πθ(yℓ∣x)πref(yℓ∣x)])]\mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[\log \sigma\!\left(\beta \left[\log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log\frac{\pi_\theta(y_\ell|x)}{\pi_{\text{ref}}(y_\ell|x)}\right]\right)\right]

DPO replaces the reinforcement-learning loop of RLHF with a classification loss on preference pairs. Under the Bradley–Terry preference model it optimizes the same KL-constrained objective (Rafailov et al., 2023).

KL-regularized RLHF objective:

max⁡πEy∼π(⋅∣x)[r(x,y)]−β⋅KL(π(⋅∣x)∥πref(⋅∣x))\max_{\pi} \mathbb{E}_{y \sim \pi(\cdot|x)}[r(x,y)] - \beta \cdot \text{KL}(\pi(\cdot|x) \| \pi_{\text{ref}}(\cdot|x))

Closed-form optimal policy (Boltzmann):

π∗(y∣x)=1Z(x)πref(y∣x)exp⁡ ⁣(1βr(x,y))\pi^{*}(y|x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y|x) \exp\!\left(\frac{1}{\beta}r(x,y)\right)

DPO loss (RL-free):

Given preference pairs (x,yw,yℓ)(x, y_w, y_\ell) (winner, loser):

LDPO(θ)=−E[log⁡σ ⁣(β[log⁡πθ(yw∣x)πref(yw∣x)−log⁡πθ(yℓ∣x)πref(yℓ∣x)])]\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}\left[\log \sigma\!\left(\beta \left[\log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log\frac{\pi_\theta(y_\ell|x)}{\pi_{\text{ref}}(y_\ell|x)}\right]\right)\right]

This is logistic regression on log-probability ratios: there is no separately trained reward model and no sampling from the policy during training.

Key Papers

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafailov et al.2023NeurIPS
Read paper →

Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

Pal et al.2024arXiv
Read paper →

SimPO: Simple Preference Optimization with a Reference-Free Reward

Meng, Xia & Chen2024NeurIPS
Read paper →

Connections

Prerequisites

Maximum LikelihoodRLHF

Leads to

KTOReward HackingConstitutional AI

Where to go next

A newer notebook covers DPO, with an interactive demo.

Continue in the newer notebook
Other connections (6)

Which limitations does it address, or leave to a later idea?

  • No pairs needed: a limitation of DPO that KTO addresses. Open KTO

    KTO removes DPO's need for paired preferences: it learns from single outputs labelled desirable or undesirable, using a value function from Kahneman and Tversky's prospect theory (Ethayarajh et al., 2024).

  • Remove reward model: a limitation of Reward Hacking that DPO addresses. Open Reward Hacking

    DPO folds the reward model into the policy by rewriting the KL-regularized RL objective as a classification loss on preference pairs (Rafailov et al., 2023). This removes the separate reward model but not over-optimization: direct alignment methods also degrade when optimized too far (Rafailov et al., 2024).

Where does it break down?

  • Reward hacking: where RLHF breaks down, taken up on this page. Open RLHF

    RLHF trains a separate reward model and then optimizes against it, and the policy can exploit that model, raising its score while real quality falls (Gao et al., 2023). DPO removes the separate reward model and the RL step, fitting the policy directly to preference pairs (Rafailov et al., 2023); its implicit reward can still be over-optimized (Rafailov et al., 2024).

  • Distribution shift: where Reward Hacking breaks down, taken up on this page. Open Reward Hacking

    DPO can also over-optimize. As the policy moves away from the reference model, its implicit reward is applied to outputs unlike the preference data, and performance can degrade even before one pass over the data is complete (Rafailov et al., 2024).

Where else does the same idea appear?

  • Likelihood ratios: the same technique appears in Maximum Likelihood. Open Maximum Likelihood

    DPO is logistic regression on log-probability ratios (log πθ − log πref). That's classic maximum-likelihood machinery: fit a classifier via cross-entropy on preference-labeled data.

Which ideas are its mathematical duals?

  • RL ↔ Classification: RLHF is its mathematical dual. Open RLHF

    KL-regularized RLHF has an optimal-policy form π*(y|x) ∝ π_ref(y|x)·exp(r/β). DPO is the supervised/logistic view of fitting that same policy from preference comparisons.