Direct Preference Optimization

The source-level representative is the beta-scaled policy/reference log-ratio:

$$\hat r_\theta(x,y) = \beta\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}.$$

Setup and shapes

For a prompt $x$, $\pi_\theta(y\mid x)$ is the full sequence probability of completion $y$. In an autoregressive language model,

$$\log \pi_\theta(y\mid x) = \sum_t \log \pi_\theta(y_t\mid x,y_{<t}).$$

A preference datum is $(x,y_w,y_\ell)$. All compared completions must have positive probability under the reference policy for the log ratio to be finite.

Define the policy and reference log-odds:

$$a_\theta = \log\pi_\theta(y_w\mid x) - \log\pi_\theta(y_\ell\mid x),$$

and

$$a_{\mathrm{ref}} = \log\pi_{\mathrm{ref}}(y_w\mid x) - \log\pi_{\mathrm{ref}}(y_\ell\mid x).$$

The reference-relative margin is

$$m_\theta=a_\theta-a_{\mathrm{ref}}.$$

DPO predicts the preference probability as

$$P_\theta(y_w\succ y_\ell\mid x) = \sigma(\beta m_\theta).$$

For one hard winner/loser label, the DPO loss is

$$\mathcal L_{\mathrm{DPO}} = -\log\sigma(\beta m_\theta).$$

Let $s_\theta=\sigma(\beta m_\theta)$. The browser demo below uses a soft target $p^*$ so the finite target log-odds is visible:

$$\mathcal L_{\mathrm{soft}} = -p^*\log s_\theta -(1-p^*)\log(1-s_\theta).$$

Ordinary hard-label DPO is the edge case $p^*\to 1$, where an isolated two-response target diverges.

From KL-regularized RLHF to DPO

For a fixed prompt $x$, write $K_x(\pi)$ for the distribution-level KL from a candidate policy to the reference policy at that prompt. The direction is $\pi(\cdot\mid x)$ to $\pi_{\mathrm{ref}}(\cdot\mid x)$:

$$K_x(\pi) = \mathrm{KL}_x(\pi\|\pi_{\mathrm{ref}}).$$

Also write the expected reward as

$$R_x(\pi) = \mathbb E_{y\sim\pi(\cdot\mid x)}[r(x,y)].$$

The KL-regularized RLHF objective is to maximize

$$J_x(\pi) = R_x(\pi)-\beta K_x(\pi).$$

Its optimizer has the reweighted form

$$\pi_r(y\mid x) = \frac{\pi_{\mathrm{ref}}(y\mid x)\exp(r(x,y)/\beta)} {Z(x)}.$$

Rearrange it:

$$r(x,y) = \beta\log \frac{\pi_r(y\mid x)} {\pi_{\mathrm{ref}}(y\mid x)} + \beta\log Z(x).$$

The partition term $Z(x)$ depends on the prompt, not on which completion won. Bradley-Terry preferences use reward differences, so the prompt-only term cancels between $y_w$ and $y_\ell$.

Bradley-Terry likelihood

Bradley-Terry models a preference as

$$P(y_w\succ y_\ell\mid x) = \sigma(r(x,y_w)-r(x,y_\ell)).$$

Substituting the implicit reward gives the DPO probability. With the margin $m_\theta=a_\theta-a_{\mathrm{ref}}$ from above:

$$P_\theta(y_w\succ y_\ell\mid x) =\sigma(\beta m_\theta).$$

Maximum likelihood on preference labels gives the DPO loss. This is why DPO is a supervised objective even though it comes from a KL-regularized RLHF derivation.

What beta does

Here $\beta$ follows the DPO paper's convention: it is the coefficient on the KL penalty in the KL-regularized RLHF objective.

• Larger $\beta$ means a stronger reference anchor in the underlying RLHF objective.
• Smaller $\beta$ means a given preference probability requires a larger policy/reference log-ratio gap.
• In the DPO loss, $\beta$ also scales the sigmoid margin, so it affects optimization dynamics. Do not read "larger $\beta$" as simply "trust preferences more."

For a soft preference probability $p^*$, matching $p^*$ requires

$$m^*=\frac{\operatorname{logit}(p^*)}{\beta}.$$

Limits of the mechanism

DPO removes the explicit reward-model training stage and the RL loop. It does not remove the assumptions behind preference learning.

It still relies on a pairwise preference model such as Bradley-Terry. It only identifies rewards up to prompt-dependent constants. It needs a usable reference policy. And in an isolated two-response hard-label setting, the logistic loss can keep increasing the winner-over-loser log-ratio rather than settling at a finite target.