Reward Hacking: Overoptimizing Preference Proxies

Let one prompt $x$ have a finite set of candidate completions $y_1,\dots,y_K$.

The target, learned proxy, and proxy error are

$$u_i = u(x,y_i), \qquad \hat r_i = u_i+\varepsilon_i, \qquad \varepsilon_i=\hat r_i-u_i.$$

A KL-regularized finite-action update shifts probability toward high proxy scores and selects proxy error:

$$\pi_\beta(i\mid x) = \frac{ \pi_{\mathrm{ref}}(i\mid x)\exp(\hat r_i/\beta) }{ \sum_j \pi_{\mathrm{ref}}(j\mid x)\exp(\hat r_j/\beta) }, \qquad E(\beta) = \sum_i \pi_\beta(i\mid x)(\hat r_i-u_i).$$

In practice we do not know $u_i$, but the toy keeps it visible so we can see the failure.

A KL-regularized RLHF-style update uses the direction

$$\mathrm{KL}(\pi\|\pi_{\mathrm{ref}}) = \sum_i \pi_i \log \frac{\pi_i}{\pi_{\mathrm{ref},i}}.$$

The unpenalized-proxy policy solves

$$\max_{\pi\in\Delta_K} \sum_i \pi_i\hat r_i - \beta \sum_i \pi_i \log \frac{\pi_i}{\pi_{\mathrm{ref},i}}.$$

Smaller $\beta$ means stronger optimization pressure. The proxy objective can improve while the real target gets worse:

$$\widehat R(\beta) = \sum_i \pi_\beta(i\mid x)\hat r_i,$$

while

$$U(\beta) = \sum_i \pi_\beta(i\mid x)u_i.$$

In this toy, reward hacking is visible when $\widehat R(\beta)$ keeps increasing but $U(\beta)$ peaks and then falls.

Optimization does not merely reveal random error; it selects for candidates where the proxy error is useful to the optimizer.

Conservative score

With an ensemble, define a mean reward $\mu_i$ and disagreement $\sigma_i$. A lower-confidence score is

$$s_i(\lambda)=\mu_i-\lambda\sigma_i.$$

Optimizing this score gives

$$\pi_{\beta,\lambda}(i\mid x) = \frac{ \pi_{\mathrm{ref}}(i\mid x)\exp(s_i(\lambda)/\beta) }{ \sum_j \pi_{\mathrm{ref}}(j\mid x)\exp(s_j(\lambda)/\beta) }.$$

In the conservative setting, the selected proxy error is still measured against the unpenalized proxy mean:

$$E_{\mathrm{sel}}(\beta,\lambda) = \sum_i \pi_{\beta,\lambda}(i\mid x) (\mu_i-u_i).$$

The lower-confidence score changes which completions become attractive to the optimizer. It does not make the proxy error disappear, and it is not scored as $s_i(\lambda)-u_i$.

This can reduce concentration on uncertain proxy exploits. It does not prove $s_i(\lambda)=u_i$; it only changes which errors are attractive.

Goodhart variants near this toy

| Variant | How to read it here | | --- | --- | | Regressional Goodhart | Directly modeled: selecting high proxy scores also selects positive proxy error $\mu_i-u_i$. | | Extremal Goodhart | Suggested, not proved: the exploit starts with low reference probability and high uncertainty, a stand-in for moving away from the reward model's reliable region. | | Adversarial Goodhart | Suggested, not dynamically modeled: the fixed "proxy exploit" represents an output pattern that scores well under the evaluator while failing the intended target. | | Causal Goodhart | Not modeled here. Causal Goodhart needs the optimization process to change the data-generating or evaluation process itself. |