RLHF: Reward Modeling + KL-Regularized Policy Optimization

The two moving pieces are the preference-trained reward gap and the KL-shaped policy update:

$$\Delta_\phi(x,y_w,y_\ell) = r_\phi(x,y_w)-r_\phi(x,y_\ell), \qquad P_\phi(y_w\succ y_\ell\mid x)=\sigma(\Delta_\phi).$$ $$J_x(\pi)= \sum_y \pi(y\mid x)r_\phi(x,y) - \beta\mathrm{KL}(\pi(\cdot\mid x)\|\pi_{\mathrm{ref}}(\cdot\mid x)), \qquad \pi^*(y\mid x)= \frac{\pi_{\mathrm{ref}}(y\mid x)\exp(r_\phi(x,y)/\beta)} {\sum_{y'}\pi_{\mathrm{ref}}(y'\mid x)\exp(r_\phi(x,y')/\beta)}.$$

The rest of this section unpacks those two witnesses.

Preference data and reward-model likelihood

Let a preference datum be $(x,y_w,y_\ell)$, where $y_w$ is preferred to $y_\ell$ for prompt $x$. Define the reward gap

$$\Delta = r_\phi(x,y_w)-r_\phi(x,y_\ell).$$

A reward model $r_\phi(x,y)\in\mathbb R$ predicts

$$P_\phi(y_w\succ y_\ell\mid x)=\sigma(\Delta).$$

The negative log-likelihood is

$$\mathcal L_{\mathrm{RM}}(\phi) = -\frac{1}{N} \sum_{i=1}^N \log\sigma(\Delta_i),$$

where

$$\Delta_i = r_\phi(x_i,y_{w,i}) - r_\phi(x_i,y_{\ell,i}).$$

Only reward differences inside the same prompt are observed. Therefore $r_\phi(x,y)+c(x)$ gives the same pairwise probabilities as $r_\phi(x,y)$.

Reward scale also needs a convention. With a fixed logistic noise model, reward gaps are measured in that model's log-odds units. With perfectly separable hard preferences, an unregularized reward model can drive gaps toward infinity. Practical systems therefore normalize, regularize, or otherwise choose a usable reward scale before policy optimization.

KL-regularized policy optimization

For a fixed prompt $x$, reference policy $\pi_{\mathrm{ref}}$, and candidate policy $\pi$, define expected reward

$$R_x(\pi) = \sum_y \pi(y\mid x)r_\phi(x,y)$$

and the KL term

$$K_x(\pi)=\mathrm{KL}(\pi(\cdot\mid x)\|\pi_{\mathrm{ref}}(\cdot\mid x)).$$

The RLHF objective is

$$J_x(\pi)=R_x(\pi)-\beta K_x(\pi).$$

Equivalently, if

$$q_y = \log \frac{\pi(y\mid x)} {\pi_{\mathrm{ref}}(y\mid x)},$$

then

$$J_x(\pi) = \sum_y \pi(y\mid x)\{r_\phi(x,y)-\beta q_y\}.$$

Assume $\pi_{\mathrm{ref}}(y\mid x)>0$ on the candidate support. Optimize $J_x$ over the finite distribution $\pi(\cdot\mid x)$ subject to $\sum_y \pi(y\mid x)=1$. The Lagrange stationarity condition is

$$r_\phi(x,y)-\beta(q_y+1)+\lambda=0.$$

Solving for $\pi(y\mid x)$ and normalizing gives

$$\pi^*(y\mid x) = \frac{ \pi_{\mathrm{ref}}(y\mid x)\exp(r_\phi(x,y)/\beta) }{ Z(x) }.$$

Here

$$Z(x) = \sum_{y'} \pi_{\mathrm{ref}}(y'\mid x) \exp(r_\phi(x,y')/\beta).$$

Larger $\beta$ keeps the policy closer to the reference. Smaller $\beta$ lets learned reward dominate.

Rearranging gives the bridge to DPO:

$$r_\phi(x,y) = \beta\log \frac{\pi^*(y\mid x)} {\pi_{\mathrm{ref}}(y\mid x)} + \beta\log Z(x).$$

The final term depends only on the prompt, so it cancels in preference differences.

PPO is the optimizer, not the definition

In language-model RLHF, the candidate space is enormous. InstructGPT-style RLHF uses PPO to optimize the learned reward with a KL penalty to the supervised-finetuned or reference model. The clean finite-action formula above is the exact optimum of a finite-action regularized objective. PPO is a stochastic optimizer for the language-model version; it is not this closed-form update, and a parametric policy with per-token KL details need not trace the toy optimum exactly.

Reward hacking

If $r_\phi(x,y)$ differs from the real target $u(x,y)$, then low $\beta$ can concentrate the policy on outputs with high proxy reward and low true utility. KL regularization reduces this pressure but does not make the proxy correct.