Reward Hacking: Overoptimizing Preference Proxies

When an imperfect learned reward is optimized beyond the range where it was validated, probability shifts toward the reward model's errors; KL penalties, ensembles, lower-confidence-bound scores and monitoring slow this down but do not make the proxy true.

Advanced · Undergraduate mathematics · about 22 minutes

Reading map and next steps

Intuition

The idea in plain words, before the symbols.

Reward hacking occurs when optimizing a proxy reward favors outputs that perform worse on the intended target. We use one prompt and a finite set of completions, including a completion the reward model overrates. The figure checks how optimization pressure and an uncertainty penalty change the probability assigned to that completion. This notebook uses probability distributions and KL regularization.

The problem is not that optimization is mysterious. It is doing exactly what we asked: put more probability on completions with high proxy reward. The failure is that the proxy reward is not the real target. When the policy is optimized hard enough, it selects for places where the reward model is wrong in the optimizer's favor.

The mechanism is:

One useful finite-action model of reward hacking is selected proxy error. Optimization increases expected proxy reward while shifting probability mass toward outputs whose true utility is lower than the proxy says.

KL penalties, uncertainty penalties, ensembles, early stopping, fresh labels and adversarial evaluation reduce optimization pressure, expose uncertainty, or slow concentration on proxy exploits. They slow the failure down; they do not prove the proxy is the target.

The difficulty is structural. Skalse et al. (2022) call a proxy unhackable if raising its expected value can never lower the true expected value, and show that, over the set of all stochastic policies, this holds only when the proxy is equivalent to the true reward (it ranks every policy the same way) or one of the two is constant. Any other proxy can be raised by some change of policy that lowers the true reward, so the practical question is how much optimization a given proxy can bear.

Sources. Amodei et al. (2016), "Concrete Problems in AI Safety", name reward hacking as an accident risk. Gao, Schulman and Hilton (2023), "Scaling Laws for Reward Model Overoptimization", measure it for learned reward models. Skalse et al. (2022), "Defining and Characterizing Reward Gaming", give the formal definition. Manheim and Garrabrant (2018), "Categorizing Variants of Goodhart's Law", supply the taxonomy used near the end of this page.

Study prompts for this section: Intuition

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Mathematics

Definitions, assumptions and the derivation.

Equation 1 Equation 2 Equation 3

Let one prompt $x$ have a finite set of candidate completions $y_1,\dots,y_K$ .

The target, learned proxy, and proxy error are

u_i = u(x,y_i), \qquad \hat r_i = u_i+\varepsilon_i, \qquad \varepsilon_i=\hat r_i-u_i.

A KL-regularized finite-action update shifts probability toward high proxy scores and selects proxy error:

\pi_\beta(i\mid x) = \frac{ \pi_{\mathrm{ref}}(i\mid x)\exp(\hat r_i/\beta) }{ \sum_j \pi_{\mathrm{ref}}(j\mid x)\exp(\hat r_j/\beta) }, \qquad E(\beta) = \sum_i \pi_\beta(i\mid x)(\hat r_i-u_i).

In practice we do not know $u_i$ , but the toy keeps it visible so we can see the failure.

A KL-regularized RLHF-style update uses the direction

\mathrm{KL}(\pi\|\pi_{\mathrm{ref}}) = \sum_i \pi_i \log \frac{\pi_i}{\pi_{\mathrm{ref},i}}.

The unpenalized-proxy policy solves

\max_{\pi\in\Delta_K} \sum_i \pi_i\hat r_i - \beta \sum_i \pi_i \log \frac{\pi_i}{\pi_{\mathrm{ref},i}}.

Smaller $\beta$ means stronger optimization pressure. The proxy objective can improve while the real target gets worse:

\widehat R(\beta) = \sum_i \pi_\beta(i\mid x)\hat r_i,

while

U(\beta) = \sum_i \pi_\beta(i\mid x)u_i.

In this toy, reward hacking is visible when $\widehat R(\beta)$ keeps increasing but $U(\beta)$ peaks and then falls. Gao, Schulman and Hilton (2023) observe the same shape with real reward models: as a policy is optimized against a learned proxy, the score from a fixed "gold" reward model, standing in for human judgment, rises, peaks and declines. In their RL experiments a KL penalty acted like early stopping: it did not improve the gold score reached at a given distance from the initial policy, though they note this may be sensitive to hyperparameters.

Optimization does not merely reveal random error; it selects for candidates where the proxy error is useful to the optimizer.

Conservative score

With an ensemble of reward models, define the mean reward $\mu_i$ and the disagreement (standard deviation across the ensemble) $\sigma_i$ . A lower-confidence-bound score is

s_i(\lambda)=\mu_i-\lambda\sigma_i.

Optimizing this score gives

\pi_{\beta,\lambda}(i\mid x) = \frac{ \pi_{\mathrm{ref}}(i\mid x)\exp(s_i(\lambda)/\beta) }{ \sum_j \pi_{\mathrm{ref}}(j\mid x)\exp(s_j(\lambda)/\beta) }.

In the conservative setting, the selected proxy error is still measured against the unpenalized proxy mean:

E_{\mathrm{sel}}(\beta,\lambda) = \sum_i \pi_{\beta,\lambda}(i\mid x) (\mu_i-u_i).

The lower-confidence-bound score changes which completions become attractive to the optimizer, which can reduce concentration on uncertain proxy exploits. It does not make the proxy error disappear, which is why the error is still measured against $\mu_i$ and not as $s_i(\lambda)-u_i$ . Coste et al. (2024), "Reward Model Ensembles Help Mitigate Overoptimization", found that related conservative objectives (the worst case over the ensemble, or the mean minus a variance penalty) reduced overoptimization in the setup of Gao et al.

Goodhart variants near this toy

Manheim and Garrabrant (2018) distinguish four ways a proxy can fail under optimization. Here is how each relates to this toy:

Regressional Goodhart (modeled directly). Selecting for a high proxy score also selects for positive proxy error $\mu_i-u_i$ .
Extremal Goodhart (suggested, not proved). The exploit starts with low reference probability and high uncertainty, a stand-in for optimization pushing into a region where the learned relationship between proxy and target no longer holds.
Adversarial Goodhart (not modeled). This variant needs a second agent that games the metric for its own ends. The fixed "proxy exploit" here only stands in for an output that scores well with the evaluator while failing the intended target.
Causal Goodhart (not modeled). When the proxy is linked to the goal only indirectly, intervening on the proxy can change the relationship between them. The scores in this toy are fixed, so it has no such mechanism.

Study prompts for this section: Mathematics

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Code

The same calculation as runnable code, in the notation of the derivation.

Code example 1import numpy as np def normalize(weights): weights = np.asarray(weights, dtype=float) return...python

This code computes the same finite-action example as the math and the demo.

import numpy as np

def normalize(weights):
    weights = np.asarray(weights, dtype=float)
    return weights / weights.sum()

def kl(p, q):
    return float(np.sum(p * (np.log(p) - np.log(q))))

def expected(p, values):
    return float(np.sum(p * values))

labels = [
    "clear answer",
    "safe refusal",
    "thin answer",
    "proxy exploit",
]

def show(p):
    return {label: round(float(x), 3) for label, x in zip(labels, p)}

# One prompt, four possible completions.
# Shapes: all arrays are (K,).
pi_ref = np.array([0.28, 0.22, 0.42, 0.08])

# True utility is hidden in practice; visible here for diagnosis.
u = np.array([1.20, 0.70, -0.30, -0.80])

# Mean reward model score. The final candidate is a reward-model exploit:
# it is truly bad, but the proxy scores it highly.
mu = np.array([1.10, 0.60, -0.10, 1.90])

# Ensemble disagreement / uncertainty.
sigma = np.array([0.15, 0.15, 0.20, 1.00])

def policy(beta, lam=0.0):
    score = mu - lam * sigma
    logits = np.log(pi_ref) + score / beta
    logits = logits - logits.max()
    return normalize(np.exp(logits))

u_ref = expected(pi_ref, u)

print("reference true utility:", round(u_ref, 3))
print()

for beta in [3.0, 2.0, 1.0, 0.7, 0.45, 0.30]:
    pi = policy(beta, lam=0.0)
    print("beta:", beta)
    print("  policy:       ", show(pi))
    print("  proxy reward: ", round(expected(pi, mu), 3))
    print("  true utility: ", round(expected(pi, u), 3))
    print("  selected err: ", round(expected(pi, mu - u), 3))
    print("  KL to ref:    ", round(kl(pi, pi_ref), 3))
    print()

print("With an uncertainty penalty:")
for beta in [0.7, 0.45, 0.30]:
    pi = policy(beta, lam=1.0)
    print("beta:", beta)
    print("  policy:       ", show(pi))
    print("  proxy reward: ", round(expected(pi, mu), 3))
    print("  true utility: ", round(expected(pi, u), 3))
    print("  KL to ref:    ", round(kl(pi, pi_ref), 3))
    print()

Study prompts for this section: Code

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Demo

Change an input, predict the result, and see which quantities respond.

ExploreChange one control at a time and name what responds.An optional observation guide follows the demo.

Explore Reward Hacking: Overoptimizing Preference Proxies

Loading the interactive demo…

Observation guide

Optional and ungraded. Opening a guide saves nothing.

Choose what to inspect in Reward Hacking: Overoptimizing Preference Proxies, then open its guide.

The demo shows one prompt's probability mass moving. Lower $\beta$ increases optimization pressure. Turning on the proxy gap makes one bad completion look excellent to the reward model. Increasing the uncertainty penalty slows concentration on that exploit, but it does not make the proxy identical to true utility.

Study prompts for this section: Demo

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Focus on

In focus

Concept: Reward Hacking: Overoptimizing Preference Proxies

What is the smallest example of Reward Hacking you can work through by hand, and what does it show?

BeforeRLHF: Reward Modeling + KL-Regularized Policy OptimizationNextProcess Reward Models: Step-Level Verifiers for Reasoning

DetailsAlignment

About this itemAlignment

6 sources citedChecking this browser

Reference id

concept:alignment/reward-hacking

Math3 equationsu_i = u(x,y_i), \qquad \hat r_i = u_i+\varepsilon_i, \qquad \varepsilon_i=\hat r_i-u_i.CodeReward Hacking: Overoptimizing Preference Proxies code example 1weights = np.asarray(weights, dtype=float)DemoReward Hacking: Overoptimizing Preference Proxies interactive demoChange one control at a time and name what responds.

After the first pass

Go deeper

Optional material for a second reading. Sources are always shown; the other panels open from their headings or from these links, and closing one keeps what you wrote or revealed.

What to look for in each sectionChoose a section and what you expect to see in it, then read short advice.

Section by section

Choose what to look for in each section

Choose what to look for (no demo yet)01 / Intuition

Illustration of a proxy reward path climbing the wrong hill while missing the true intended goal.

What to look for

Start with the picture, metaphor, or geometric mechanism.

Choose first

Choose what you expect to see in this section of Reward Hacking: Overoptimizing Preference Proxies.

Questions to ask of the pictureChoose a part of the picture and what you expect, then read short advice.

Looking at the picture

Questions to ask of the picture

4 of 4 sections writtenNo live demo yet

Visual cueWhich part of the picture should you look at first?

Highlight size2/4

Question

Which part of the picture should you look at first?

Choose first

Pick the part of the picture you expect to explain Reward Hacking: Overoptimizing Preference Proxies.

Sources

References for this page

A reference shows where an idea comes from; it does not vouch for every step on this page.

paper · 2016Concrete Problems in AI SafetyAmodei et al.

Why it is cited

arXiv preprint. Lists avoiding reward hacking among five practical research problems on accident risk: an agent games its objective in ways the designer did not intend.

Open source

paper · 2023Scaling Laws for Reward Model OveroptimizationGao, Schulman, and Hilton

Why it is cited

ICML 2023. Optimizes policies against a proxy reward model trained on labels from a fixed gold reward model: the gold score rises, peaks and falls, with different curves for best-of-n and RL. A KL penalty acted like early stopping.

Open source

paper · 2018Categorizing Variants of Goodhart's LawManheim and Garrabrant

Why it is cited

arXiv preprint. Defines the regressional, extremal, causal and adversarial variants of Goodhart's law used in the list near the end of this page.

Open source

paper · 2017Deep reinforcement learning from human preferencesChristiano et al.

Why it is cited

NeurIPS 2017. Learns a reward model from human comparisons, the setting in which a learned reward can be over-optimized.

Open source

paper · 2022Defining and Characterizing Reward GamingSkalse et al.

Why it is cited

NeurIPS 2022; arXiv title 'Defining and Characterizing Reward Hacking'. A proxy is unhackable if raising its expected value never lowers the true one; over all stochastic policies, that requires equivalent rewards or a constant one.

Open source

paper · 2024Reward Model Ensembles Help Mitigate OveroptimizationCoste et al.

Why it is cited

ICLR 2024. In the setup of Gao et al., conservative ensemble objectives (worst case, or mean minus a variance penalty) reduced overoptimization under best-of-n and PPO.

Open source

What each claim rests on1 claim, each with its sources, where it appears on this page, and its caveats.

Claims and sources

When an imperfect learned reward is optimized beyond the range where it was validated, probability shifts toward the reward model's errors; KL penalties, ensembles, lower-confidence-bound scores and monitoring slow this down but do not make the proxy true.

Checked against the cited passages

The checks were made by this site, not by an independent reviewer.

Reward hacking can be modeled as selected proxy error: stronger optimization can raise expected learned reward by shifting probability toward outputs where the proxy overestimates true utility; a KL penalty and ensemble uncertainty penalties can slow this shift but do not make the proxy the target.

Sources

Concrete Problems in AI SafetyarXiv preprint. Lists avoiding reward hacking among five practical research problems on accident risk: an agent games its objective in ways the des...Scaling Laws for Reward Model OveroptimizationICML 2023. Optimizes policies against a proxy reward model trained on labels from a fixed gold reward model: the gold score rises, peaks and falls,...

What the sources support

Amodei et al. describe reward hacking as an agent gaming its objective. Gao et al. show that optimizing an imperfect reward model keeps raising the proxy score while a gold-standard score peaks and then falls.

Where to see it on this page

Concrete Problems in AI Safety Scaling Laws for Reward Model Overoptimization Equation 1 Equation 2 Code example 1 Demo

Equation 1

u_i = u(x,y_i), \qquad \hat r_i = u_i+\varepsilon_i, \qquad \varepsilon_i=\hat r_i-u_i.

Equation 2

\pi_\beta(i\mid x) = \frac{ \pi_{\mathrm{ref}}(i\mid x)\exp(\hat r_i/\beta) }{ \sum_j \pi_{\mathrm{ref}}(j\mid x)\exp(\hat r_j/\beta) }, \qquad E(\beta) = \sum_i \pi_\beta(i\mid x)(\hat r_i-u_i).

Caveat

A small finite example, not a measurement. Gao et al. found a KL penalty acted like early stopping and note this may depend on hyperparameters. The uncertainty penalty here slows the shift; it is not shown t...

StatusChecked against the cited passages.

Amodei et al. describe reward hacking as an agent gaming a wrong or exploitable objective. Gao et al. show reward-model overoptimization in RLHF: optimizing a proxy can raise the learned reward while the gold reward falls. The finite example on this page illustrates selected proxy error, the KL-regularized shift of probability and the slowing effect of an uncertainty penalty.

Explain it without the pageWrite your own explanation before asking for feedback. Your draft and any hints stay while this page is open.

Practice · Reward Hacking: Overoptimizing Preference Proxies

Try the idea in your own words

Concept · Selected for practice

Reward Hacking: Overoptimizing Preference Proxies

What it rests on: Sources: Concrete Problems in AI Safety; Scaling Laws for Reward Model Overoptimization; Categorizing Variants of Goodhart's Law

Context and links

Alignment

Choose a task

Explain the mechanism

For Reward Hacking: Overoptimizing Preference Proxies: What is the smallest example of Reward Hacking you can work through by hand, and what does it show? Explain your answer, including what changes, why, and which assumption matters.

Your answerNo answer yet

A rough first thought is enough. Your draft stays when you change tasks.

Your draft stays on this page and is cleared when you leave or reload.

A little help · Explain

Open one hint at a time. These are suggestions, not your answer or a grade.

0 of 3 hints shown for this question.

Clearing your answer keeps this help record. Outside help is your own report; this page cannot check it.

Where am I stuck? (optional)

Your own description, not an automatic diagnosis

Choose one, or leave this unspecified. Select it again to clear it.

Take your draft to a feedback conversation

No AI feedback runs here. You can copy a prompt to use elsewhere; nothing is sent automatically. Review the text before sharing, and leave out private information.

Write an attempt before copying a feedback prompt.

A draft, or an AI reply to it, is not a test of what you have learned. To check that, try a different case later without help.

Ask about this pagePick one item on this page and copy a prompt about it into an AI assistant you already use. Notes you write here stay in this browser.

Ask about this pageClose

ConceptReward Hacking: Overoptimizing Preference ProxiesSources: Concrete Problems in AI Safety; Scaling Laws for Reward Model Overoptimization; Categorizing Variants of Goodhart's Lawweights = np.asarray(weights, dtype=float)

Your question

Choose what your question is about

Pick the idea, equation, source, code, claim, misconception or demo state you want to ask about. The prompt you copy includes it, so the answer can stay on that item.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptAlignment

Reward Hacking: Overoptimizing Preference Proxies

Starting question

What is the smallest example of Reward Hacking you can work through by hand, and what does it show?

SourcesCheck the 6 cited sources listed under Sources.Notes can be saved for this item

Points of view on this item

Fixed prompts generated from this item. They are not comments from people or independent reviews.

Learner’s first stepAsk what would make "Reward Hacking: Overoptimizing Preference Proxies" feel predictable rather than familiar.

Assumption

The 6 cited sources must support this exact item, not just the surrounding topic.

Source-checking summary

Connect the definition to one equation, piece of code or demo before widening the discussion.

Proposed experiment

Change one input, then check that the same relationship holds in the mathematics, the code and the demo.

Next action

You can state the mechanism in your own words

Evidence4 checks

ObservationChecking for a saved observation

ActionReady for one action

PromptLearner prompt ready to copy

Go to this item

01ObservationChecking what this browser has saved

02EvidenceChecking for a saved observation

03SourcesCheck the 6 cited sources listed under Sources.

04Next stepSave one next action

Local action draftNo local draft saved yetOpen when you are ready to write one next action

Local action draft

This draft stays in this browser and is attached to this item only.

Draft noteNext action

No local draft saved.

Evidence to inspect

What the 6 cited sources say about this exact item
The definition, its prerequisites, and a contrasting concept
The equation or code that makes the concept concrete
One demo state that shows what stays the same as the inputs change

What would resolve this

You can state the mechanism in your own words
You can name the prerequisite that would clear up the confusion
You can predict how the result changes when one input changes

Reference id

concept/concept-notebook/alignment/reward-hacking
concept:alignment/reward-hacking

Return to reading

Reading map

In this concept

Change one control at a time and name what responds.

Try the demo

Continue the curriculum

Part VII · unit 4 of 7

Process Reward Models: Step-Level Verifiers for Reasoning

Connections and sources

Domain

Alignment

alignment
reward-hacking
goodhart
overoptimization

Prerequisites

RLHF: Reward Modeling + KL-Regularized Policy Optimization

Leads to

Process Reward Models: Step-Level Verifiers for Reasoning

Study aid

Ask an assistant about this notebook

Compose a question with this notebook as context, then copy it into an AI assistant you already use.

Compose a question

Your question

GoalComfortStyleStuck on

Copy-only tool: this panel composes a prompt from this page and your choices above, for an AI assistant you already use. Nothing you enter here is sent from this page.