Continuous Function

Foundation reference

Reward Hacking & Overoptimization: Goodhart's Law in Preference Optimization

Reward hacking occurs when a policy raises a proxy reward without improving the quality that reward was meant to measure. In Gao et al. (2023), the proxy was a learned reward model and a larger 'gold' model stood in for human judgments; as the policy moved away from its starting point, the gold reward rose and then fell while the proxy reward kept rising.

A newer notebook covers this concept. This older page keeps its summary, math and sources.

Concept 26 of 100Scaling & AlignmentStudy phase 7
Open the demoOpen the newer notebookAll concepts
Key equation
rUWO(x,y)=μ(x,y)−λ σ2(x,y)r_{\text{UWO}}(x,y)=\mu(x,y)-\lambda\,\sigma^2(x,y)
Reading map and next steps

Study guide

Where to start

Concept 26 of 100Reward HackingScaling & Alignment · Study phase 7: Alignment & RLHF
Question

What does Goodhart's law ("when a measure becomes a target, it ceases to be a good measure") predict for a policy trained to maximize a learned reward?

Open the demoGo to the mathOpen the newer notebookNext concept
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.

Before you move a control in the demo, say what you expect to change.

CheckCheck against the math and the papers.

Compare what the demo shows with the key equation and with the papers cited below.

GeneralizeName what stays the same.

Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.

ContinueContinue in the newer notebook.

The newer notebook covers the same concept and has an interactive demo.

Why It Matters for Modern Models

  • Reward hacking occurs when a policy raises a proxy reward without improving the quality that reward was meant to measure. In Gao et al. (2023), the proxy was a learned reward model and a larger 'gold' model stood in for human judgments; as the policy moved away from its starting point, the gold reward rose and then fell while the proxy reward kept rising.
  • A KL penalty slows reward hacking but does not remove it. In Gao et al.'s experiments it behaved much like early stopping: it limited how far the policy moved but did not raise the true reward reached at a given distance.
  • Offline methods such as DPO are affected too: with sparse or noisy preference data, poorly covered responses can be overvalued (Type I) and good responses undervalued (Type II) (Rashidinejad & Tian, 2025).
  • Disagreement within a reward-model ensemble is a usable warning sign: a high average reward with high disagreement marks a response the reward models cannot vouch for (Coste et al., 2024).
  • Retraining the reward model on new data over several rounds of RLHF reduces overoptimization from round to round, but the gains level off (Wolf, Kirk & Musolesi, 2025).

Questions to Think About

  • Why is reward hacking usually a sign of distribution shift, with the policy producing outputs the reward model never saw, rather than of a model "trying" to cheat?
  • If a preference dataset contains no examples of a safety-critical case, why can an offline method such as DPO not learn the right behavior for it?
  • How do sparse or noisy preference labels make a poor response look good (Type I) or a good response look bad (Type II)?
  • Why might a high mean reward together with high disagreement across a reward-model ensemble mark a region where the proxy reward has broken down?

Interactive Demo

Core Math

If you prefer intuition first, try the demo above, then come back for the math.

Key Equation
rUWO(x,y)=μ(x,y)−λ σ2(x,y)r_{\text{UWO}}(x,y)=\mu(x,y)-\lambda\,\sigma^2(x,y)

Reward hacking happens when a policy is optimized against a proxy reward, usually a reward model r^ϕ\hat r_\phi learned from a finite set of human preferences, and finds outputs where the proxy is high but the true quality is not. Two things make this likely: the policy produces outputs unlike those the reward model was trained on (distribution shift), and the preference labels were noisy to begin with.

Proxy reward with a KL penalty (the RLHF objective):

max⁡πθEy∼πθ(⋅∣x)[r^ϕ(x,y)]−β KL ⁣(πθ(⋅∣x)∥πref(⋅∣x))\max_{\pi_\theta} \mathbb{E}_{y\sim \pi_\theta(\cdot|x)}[\hat r_\phi(x,y)] - \beta\,\text{KL}\!\left(\pi_\theta(\cdot|x) \| \pi_{\text{ref}}(\cdot|x)\right)

The KL term keeps the policy near the reference model πref\pi_{\text{ref}}, where the reward model is more trustworthy. In a synthetic setup where a larger "gold" reward model stands in for human judgement, Gao et al. (2023) found that as the policy moves away from the reference, the proxy reward keeps rising while the gold reward first rises and then falls.

Conservative optimization with a reward-model ensemble:

Train KK reward models rϕ1,…,rϕKr_{\phi_1}, \dots, r_{\phi_K} and compute their mean and variance:

μ(x,y)=1K∑j=1Krϕj(x,y),σ2(x,y)=1K∑j=1K(rϕj(x,y)−μ(x,y))2\mu(x,y)=\frac{1}{K}\sum_{j=1}^K r_{\phi_j}(x,y), \quad \sigma^2(x,y)=\frac{1}{K}\sum_{j=1}^K\left(r_{\phi_j}(x,y)-\mu(x,y)\right)^2

Uncertainty-weighted optimization (UWO) maximizes

rUWO(x,y)=μ(x,y)−λ σ2(x,y)r_{\text{UWO}}(x,y)=\mu(x,y)-\lambda\,\sigma^2(x,y)

and worst-case optimization maximizes min⁡jrϕj(x,y)\min_j r_{\phi_j}(x,y) (Coste et al., 2024). Both prefer outputs on which the reward models agree.

InfoRM: an information bottleneck in the reward model:

max⁡ϕE[log⁡pϕ(ℓ∣z)]−α KL ⁣(qϕ(z∣x,y)∥p(z))\max_{\phi} \mathbb{E}[\log p_\phi(\ell|z)] - \alpha\,\text{KL}\!\left(q_\phi(z|x,y) \| p(z)\right)

where ℓ\ell is the preference label and zz is a compressed representation of the prompt and response. The KL term limits how much information zz keeps, which aims to discard features that happen to predict the training labels but are unrelated to what people prefer (Miao et al., 2024).

Key Papers

Scaling Laws for Reward Model Overoptimization

Gao, Schulman & Hilton2023ICML
Read paper →

Reward Model Ensembles Help Mitigate Overoptimization

Coste et al.2024ICLR
Read paper →

InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling

Miao et al.2024NeurIPS
Read paper →

Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking

Rashidinejad & Tian2025ICLR
Read paper →

Reward Model Overoptimisation in Iterated RLHF

Wolf, Kirk & Musolesi2025arXiv
Read paper →

Connections

Prerequisites

RLHFDPOKTO

Where to go next

A newer notebook covers Reward Hacking, with an interactive demo.

Continue in the newer notebook
Other connections (5)

Which limitations does it address, or leave to a later idea?

  • Remove reward model: a limitation of Reward Hacking that DPO addresses. Open DPO

    DPO folds the reward model into the policy by rewriting the KL-regularized RL objective as a classification loss on preference pairs (Rafailov et al., 2023). This removes the separate reward model but not over-optimization: direct alignment methods also degrade when optimized too far (Rafailov et al., 2024).

Where does it break down?

  • Distribution shift: where Reward Hacking breaks down, taken up in DPO. Open DPO

    DPO can also over-optimize. As the policy moves away from the reference model, its implicit reward is applied to outputs unlike the preference data, and performance can degrade even before one pass over the data is complete (Rafailov et al., 2024).

  • Proxy gaming: where RLHF breaks down, taken up on this page. Open RLHF

    RLHF optimizes a learned reward model, which is only a proxy for human preferences. Optimized hard enough, the policy finds outputs that the reward model scores highly but people would not: the proxy reward keeps rising while a gold-standard reward first rises and then falls (Gao et al., 2023).

  • Smaller reward models: where Efficiency breaks down, taken up on this page. Open Efficiency

    Cheaper alignment pipelines can be more exposed to reward hacking. Gao et al. (2023) found that smaller proxy reward models are over-optimized more readily than larger ones, so shrinking the reward model to save compute leaves less room to optimize against it safely.

Where else does the same idea appear?

  • Learned loss gaming: the same technique appears in GANs. Open GANs

    GANs train a generator against a learned discriminator; RLHF trains a policy against a learned reward model. Both create adversarial dynamics where the generator/policy can exploit weaknesses in the learned evaluator.