Reward Hacking: Overoptimizing Preference Proxies

When an imperfect preference proxy is optimized past its validation regime, policy mass shifts toward reward-model errors; KL, ensembles, LCBs, and monitoring slow this down but do not make the proxy true.

published · difficulty 4/5 · 22 min read

Intuition

Build the mental picture first so the rest of the page has something to attach to.

PredictName the object in plain language, then predict what should change.Leave with one reusable mental picture before notation appears.

Source map: Amodei et al. introduce reward hacking as a wrong-objective safety problem; Gao et al. study reward-model overoptimization in RLHF; Manheim and Garrabrant provide the Goodhart-variants taxonomy used near the end of this page.

A reward model can validate well on held-out comparisons. Why can the policy get worse when we keep optimizing it?

The problem is not that optimization is mysterious. It is doing exactly what we asked: put more probability on completions with high proxy reward. The failure is that the proxy reward is not the real target. When the policy is optimized hard enough, it selects for places where the reward model is wrong in the optimizer's favor.

The mechanism is:

One useful finite-action model of reward hacking is selected proxy error. Optimization increases expected proxy reward while shifting probability mass toward outputs whose true utility is lower than the proxy says.

KL anchors, uncertainty penalties, ensembles, early stopping, fresh labels, and adversarial evaluation are brakes: in this setup they reduce optimization pressure, expose uncertainty, or slow concentration on proxy exploits. They do not prove the proxy is the target.

Section prompt

Copy-only prompts — each action copies a page-grounded prompt to your clipboard. Nothing is sent by this site.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

InspectTrack the same object through the notation and check each symbol.Leave with the invariant the equations preserve.

Equation 1Inspect equation objectu_i = u(x,y_i), \qquad \hat r_i = u_i+\varepsilon_i, \qquad \varepsilon_i=\hat r_i-u_i.Equation 2Inspect equation object\pi_\beta(i\mid x) = \frac{ \pi_{ref}(i\mid x)\exp(\hat r_i/\beta) }{ \sum_j \pi_{ref}(j\mid...Equation 3Inspect equation objectKL(\pi\|\pi_{ref}) = \sum_i \pi_i \log (\pi_i)/(\pi_{ref,i)}.

Let one prompt $x$ have a finite set of candidate completions $y_1,\dots,y_K$ .

The target, learned proxy, and proxy error are

u_i = u(x,y_i), \qquad \hat r_i = u_i+\varepsilon_i, \qquad \varepsilon_i=\hat r_i-u_i.

A KL-regularized finite-action update shifts probability toward high proxy scores and selects proxy error:

\pi_\beta(i\mid x) = \frac{ \pi_{\mathrm{ref}}(i\mid x)\exp(\hat r_i/\beta) }{ \sum_j \pi_{\mathrm{ref}}(j\mid x)\exp(\hat r_j/\beta) }, \qquad E(\beta) = \sum_i \pi_\beta(i\mid x)(\hat r_i-u_i).

In practice we do not know $u_i$ , but the toy keeps it visible so we can see the failure.

A KL-regularized RLHF-style update uses the direction

\mathrm{KL}(\pi\|\pi_{\mathrm{ref}}) = \sum_i \pi_i \log \frac{\pi_i}{\pi_{\mathrm{ref},i}}.

The unpenalized-proxy policy solves

\max_{\pi\in\Delta_K} \sum_i \pi_i\hat r_i - \beta \sum_i \pi_i \log \frac{\pi_i}{\pi_{\mathrm{ref},i}}.

Smaller $\beta$ means stronger optimization pressure. The proxy objective can improve while the real target gets worse:

\widehat R(\beta) = \sum_i \pi_\beta(i\mid x)\hat r_i,

while

U(\beta) = \sum_i \pi_\beta(i\mid x)u_i.

In this toy, reward hacking is visible when $\widehat R(\beta)$ keeps increasing but $U(\beta)$ peaks and then falls.

Optimization does not merely reveal random error; it selects for candidates where the proxy error is useful to the optimizer.

Conservative score

With an ensemble, define a mean reward $\mu_i$ and disagreement $\sigma_i$ . A lower-confidence score is

s_i(\lambda)=\mu_i-\lambda\sigma_i.

Optimizing this score gives

\pi_{\beta,\lambda}(i\mid x) = \frac{ \pi_{\mathrm{ref}}(i\mid x)\exp(s_i(\lambda)/\beta) }{ \sum_j \pi_{\mathrm{ref}}(j\mid x)\exp(s_j(\lambda)/\beta) }.

In the conservative setting, the selected proxy error is still measured against the unpenalized proxy mean:

E_{\mathrm{sel}}(\beta,\lambda) = \sum_i \pi_{\beta,\lambda}(i\mid x) (\mu_i-u_i).

The lower-confidence score changes which completions become attractive to the optimizer. It does not make the proxy error disappear, and it is not scored as $s_i(\lambda)-u_i$ .

This can reduce concentration on uncertain proxy exploits. It does not prove $s_i(\lambda)=u_i$ ; it only changes which errors are attractive.

Goodhart variants near this toy

| Variant | How to read it here | | --- | --- | | Regressional Goodhart | Directly modeled: selecting high proxy scores also selects positive proxy error $\mu_i-u_i$ . | | Extremal Goodhart | Suggested, not proved: the exploit starts with low reference probability and high uncertainty, a stand-in for moving away from the reward model's reliable region. | | Adversarial Goodhart | Suggested, not dynamically modeled: the fixed "proxy exploit" represents an output pattern that scores well under the evaluator while failing the intended target. | | Causal Goodhart | Not modeled here. Causal Goodhart needs the optimization process to change the data-generating or evaluation process itself. |

Section prompt

Copy-only prompts — each action copies a page-grounded prompt to your clipboard. Nothing is sent by this site.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

TraceMatch variables to symbols before reading the implementation.Leave with a runnable witness for the math.

Code witness 1import numpy as np def normalize(weights): weights = np.asarray(weights, dtype=float) return...python

This witness implements the same finite-action object as the math and demo.

import numpy as np

def normalize(weights):
    weights = np.asarray(weights, dtype=float)
    return weights / weights.sum()

def kl(p, q):
    return float(np.sum(p * (np.log(p) - np.log(q))))

def expected(p, values):
    return float(np.sum(p * values))

labels = np.array([
    "clear answer",
    "safe refusal",
    "thin answer",
    "proxy exploit",
])

# One prompt, four possible completions.
# Shapes: all arrays are (K,).
pi_ref = np.array([0.28, 0.22, 0.42, 0.08])

# True utility is hidden in practice; visible here for diagnosis.
u = np.array([1.20, 0.70, -0.30, -0.80])

# Mean reward model score. The final candidate is a reward-model exploit:
# it is truly bad, but the proxy scores it highly.
mu = np.array([1.10, 0.60, -0.10, 1.90])

# Ensemble disagreement / uncertainty.
sigma = np.array([0.15, 0.15, 0.20, 1.00])

def policy(beta, lam=0.0):
    score = mu - lam * sigma
    logits = np.log(pi_ref) + score / beta
    logits = logits - logits.max()
    return normalize(np.exp(logits))

u_ref = expected(pi_ref, u)

print("reference true utility:", round(u_ref, 3))
print()

for beta in [3.0, 2.0, 1.0, 0.7, 0.45, 0.30]:
    pi = policy(beta, lam=0.0)
    print("beta:", beta)
    print("  policy:       ", dict(zip(labels, np.round(pi, 3))))
    print("  proxy reward: ", round(expected(pi, mu), 3))
    print("  true utility: ", round(expected(pi, u), 3))
    print("  selected err: ", round(expected(pi, mu - u), 3))
    print("  KL to ref:    ", round(kl(pi, pi_ref), 3))
    print()

print("With an uncertainty penalty:")
for beta in [0.7, 0.45, 0.30]:
    pi = policy(beta, lam=1.0)
    print("beta:", beta)
    print("  policy:       ", dict(zip(labels, np.round(pi, 3))))
    print("  proxy reward: ", round(expected(pi, mu), 3))
    print("  true utility: ", round(expected(pi, u), 3))
    print("  KL to ref:    ", round(kl(pi, pi_ref), 3))
    print()

Section prompt

Copy-only prompts — each action copies a page-grounded prompt to your clipboard. Nothing is sent by this site.

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

ManipulateChange one control and predict the visible response before reveal.Leave with the observed invariant or a repaired model.

Live Concept Demo

Explore Reward Hacking: Overoptimizing Preference Proxies

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5undergraduatecode-aligned

Demo inquiry checkpoint

Manipulate one control and predict the visible change.

01Choose lensTrace a quantity

02ObserveDemo state pending

03GroundName the equation, invariant, or control that explains it.

04CarryNext: Process Reward Models: Step-Level Verifiers for Reasoning

Choose what to inspect in Reward Hacking: Overoptimizing Preference Proxies. This shared fallback is an observation guide, not evidence of learning.

Loading interactive demo...

Use the demo to watch one prompt's probability mass move. Lower $\beta$ increases optimization pressure. Turning on the proxy gap makes one bad completion look excellent to the reward model. Increasing the uncertainty penalty slows concentration on that exploit, but it does not make the proxy identical to true utility.

Section prompt

Copy-only prompts — each action copies a page-grounded prompt to your clipboard. Nothing is sent by this site.

Object

4/4 sections ready

Concept: Reward Hacking: Overoptimizing Preference Proxies

What is the smallest example that makes Reward Hacking: Overoptimizing Preference Proxies click without losing the math?

BeforeRLHF: Reward Modeling + KL-Regularized Policy OptimizationNow4/4 sections readyTryManipulate one control and predict the visible change.NextProcess Reward Models: Step-Level Verifiers for Reasoning

Object contextAlignment

StateAlignment

4 sources attachedChecking local snapshot

concept:alignment/reward-hacking

Math3 equation objectsu_i = u(x,y_i), \qquad \hat r_i = u_i+\varepsilon_i, \qquad \varepsilon_i=\hat r_i-u_i.CodeReward Hacking: Overoptimizing Preference Proxies code witness 1weights = np.asarray(weights, dtype=float)DemoReward Hacking: Overoptimizing Preference Proxies predictionManipulate one control and predict the visible change.

ConceptLearner lens

Reward Hacking: Overoptimizing Preference Proxies

What is the smallest example that makes Reward Hacking: Overoptimizing Preference Proxies click without losing the math?

Mode questionCan I say the mechanism back in one sentence before I reveal anything?

Start with the prediction checkpoint, then compare the reveal to the mental model.

Take this move

Study modes

Keep the object fixed; change the lens.

01PredictCommit before reveal.02ExplainName the invariant.03Ground4 source links nearby.04ExperimentChange one knob.05CarrySave the next question.

Route back through the notebook

Carry the same object through intuition, math, code, and demo.

4/4 sections ready

Carry inRLHF: Reward Modeling + KL-Regularized Policy Optimization

Bring the mental model from RLHF: Reward Modeling + KL-Regularized Policy Optimization; this page will reuse it instead of restarting from zero.

Work hereReward Hacking: Overoptimizing Preference Proxies

Carry outProcess Reward Models: Step-Level Verifiers for Reasoning

The next edge should feel earned: use the demo prediction here before following Process Reward Models: Step-Level Verifiers for Reasoning.

After The First Pass

Turn the concept into an inspected object.

The lower panels are one second act: keep the object fixed, inspect it visually, check source boundaries, practice transfer, then attach the research question.

ConceptReward Hacking: Overoptimizing Preference ProxiesAlignment

01MechanismInspect motionSee how the idea changes form before adding more explanation.02VisualChoose a cueBind one visible mark to one mathematical role.03SourcesCheck boundary4 source links attached to the object.04ClaimAudit evidenceTreat the page claim as something to verify.05PracticeAttempt transferWrite before feedback so the model has something to repair.06RoomAsk groundedAttach the question to the same exact object.

Mechanism Storyboard

See the idea move before the page explains it

Demo notes open01 / Intuition

Editorial alignment illustration of a proxy reward path climbing the wrong hill while missing the true intended goal.

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Reward Hacking: Overoptimizing Preference Proxies should make visible.

Visual Inquiry

Make the image answer a mathematical question

4/4 stages readyDemo notes connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Reward Hacking: Overoptimizing Preference Proxies easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

Object - ConceptReward Hacking: Overoptimizing Preference ProxiesQuestion

What is the smallest example that makes Reward Hacking: Overoptimizing Preference Proxies click without losing the math?

concept:alignment/reward-hacking

Boundary

sources: amodei-2016-concrete-safety, gao-2022-reward-overoptimization, manheim-2018-goodhart

Check

Open the closest source note before trusting the local explanation.

Evidence

4 selected-object sources shown first; 4 references total.

Next move

Audit the claim boundary, then ask from the same selected object.

selected object source · paper · 2016Concrete Problems in AI SafetyAmodei et al.

Located CF editorial boundary

Names reward hacking as an accident-risk problem caused by optimizing the wrong objective proxy.

Used here as

Amodei grounds reward hacking as an objective that can be gamed or pervert designer intent. Gao grounds RLHF overoptimization of an imperfect reward-model proxy: proxy optimization can ra...

Caveat

Toy finite-action witness only. Gao's KL result is mixed and hyperparameter-sensitive; the uncertainty/LCB penalty is page-local, not sourced as a sufficient mitigation. No direct human u...

Open source

selected object source · paper · 2022Scaling Laws for Reward Model OveroptimizationGao et al.

Located CF editorial boundary

Studies RLHF reward-model overoptimization: optimizing an imperfect proxy too much can reduce gold-standard reward-model score; includes KL-penalty effects.

Used here as

Amodei grounds reward hacking as an objective that can be gamed or pervert designer intent. Gao grounds RLHF overoptimization of an imperfect reward-model proxy: proxy optimization can ra...

Caveat

Toy finite-action witness only. Gao's KL result is mixed and hyperparameter-sensitive; the uncertainty/LCB penalty is page-local, not sourced as a sufficient mitigation. No direct human u...

Open source

selected object source · paper · 2018Categorizing Variants of Goodhart's LawManheim and Garrabrant

Located CF editorial boundary

Classifies Goodhart variants and explains metric/proxy overoptimization failure modes; supports the Goodhart table rather than the selected-error mechanism itself.

Used here as

The selected object cites this source; inspect the exact claim before treating it as support.

Caveat

Attached source metadata is a review boundary, not proof of the local explanation.

Open source

selected object source · paper · 2017Deep reinforcement learning from human preferencesChristiano et al.

Located CF editorial boundary

Provides the preference-learning setup where learned rewards can become overoptimized proxies.

Used here as

The selected object cites this source; inspect the exact claim before treating it as support.

Caveat

Attached source metadata is a review boundary, not proof of the local explanation.

Open source

Claim Review

When an imperfect preference proxy is optimized past its validation regime, policy mass shifts toward reward-model errors; KL, ensembles, LCBs, and monitoring slow this down but do not make the proxy true.

Object - ConceptReward Hacking: Overoptimizing Preference ProxiesQuestion

What is the smallest example that makes Reward Hacking: Overoptimizing Preference Proxies click without losing the math?

concept:alignment/reward-hacking

Boundary

sources: amodei-2016-concrete-safety, gao-2022-reward-overoptimization, manheim-2018-goodhart

Check

Treat every claim as provisional until source support and a local witness agree.

Evidence

1 structured claim check on this concept.

Next move

Run the prediction or practice transfer before asking for a grounded review.

1 CF editorial source-scope review recorded

Publisher-side editorial review is not independent replication. Claims without it still need exact source-support review. 4 references and 3 local witnesses are available for inspection.

Reward hacking can be modeled as selected proxy error: stronger optimization can raise expected learned reward by shifting probability toward outputs where the proxy overestimates true utility; KL anchoring and page-local uncertainty penalties can slow this shift but do not make the proxy the target.

Registered CF editorial boundary

Concrete Problems in AI SafetyNames reward hacking as an accident-risk problem caused by optimizing the wrong objective proxy.Scaling Laws for Reward Model OveroptimizationStudies RLHF reward-model overoptimization: optimizing an imperfect proxy too much can reduce gold-standard reward-model score; includes KL-penalty...

Used here as

Amodei grounds reward hacking as an objective that can be gamed or pervert designer intent. Gao grounds RLHF overoptimization of an imperfect reward-model proxy: proxy optimization can raise proxy reward whi...

Local witness

source-span-amodei-2016-concrete-safety source-span-gao-2022-reward-overoptimization math-object-1 math-object-2 code-witness-1 interactive-demo

Equation 1

u_i = u(x,y_i), \qquad \hat r_i = u_i+\varepsilon_i, \qquad \varepsilon_i=\hat r_i-u_i.

Equation 2

\pi_\beta(i\mid x) = \frac{ \pi_{\mathrm{ref}}(i\mid x)\exp(\hat r_i/\beta) }{ \sum_j \pi_{\mathrm{ref}}(j\mid x)\exp(\hat r_j/\beta) }, \qquad E(\beta) = \sum_i \pi_\beta(i\mid x)(\hat r_i-u_i).

Caveat

Toy finite-action witness only. Gao's KL result is mixed and hyperparameter-sensitive; the uncertainty/LCB penalty is page-local, not sourced as a sufficient mitigation. No direct human utility access, unive...

Review stateCF editorial source-scope reviewClaim metadata: source checkedPublisher-side editorial review only; not independent replication. Check caveats and exact source scope.

Amodei supports reward hacking as a wrong-objective/gamed-reward problem. Gao supports RLHF reward-model overoptimization where proxy optimization can improve learned reward while hindering gold reward. The page's finite-action math/code/demo are acceptable toy witnesses for selected proxy error, KL-softmax shift, and local uncertainty braking.

Reviewer: codex+oracle; reviewed 2026-05-07

Practice notebook

Use the idea, then test it somewhere new

AttemptNo learning claim inferred

Object - ConceptReward Hacking: Overoptimizing Preference ProxiesQuestion

What is the smallest example that makes Reward Hacking: Overoptimizing Preference Proxies click without losing the math?

concept:alignment/reward-hacking

Boundary

sources: amodei-2016-concrete-safety, gao-2022-reward-overoptimization, manheim-2018-goodhart

Check

Use one state from Reward Hacking: Overoptimizing Preference Proxies to explain what changes, why it changes, and which assumption the explanation needs.

Evidence

No learner move yet; no learning state is inferred.

Next move

Write first, use only the help you need, then try a new case without it.

Explain

Use one state from Reward Hacking: Overoptimizing Preference Proxies to explain what changes, why it changes, and which assumption the explanation needs.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Attempt context

Learner move

Write an attempt before asking the companion.

Assistance used

0 of 3 progressive hints opened.

Evidence boundary

This draft and any AI response do not establish mastery; a later unassisted case can.

Misconception check

Grounded object roomClose

Selected object routeAsk from this object; carry one invariant back.sources: amodei-2016-concrete-safety, gao-2022-reward-overoptimization, manheim-2018-goodhart

ObjectConceptReward Hacking: Overoptimizing Preference Proxies
PredictBefore revealReward Hacking: Overoptimizing Preference Proxies prediction
WitnessCompare codeReward Hacking: Overoptimizing Preference Proxies code witness 1
RoomAsk groundedChecking local snapshot

ConceptReward Hacking: Overoptimizing Preference ProxiesAlignment

Code witness comparisonReward Hacking: Overoptimizing Preference Proxies code witness 1weights = np.asarray(weights, dtype=float)Prediction before revealReward Hacking: Overoptimizing Preference Proxies predictionManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes Reward Hacking: Overoptimizing Preference Proxies click without losing the math?Checking local snapshot

Research Room

Attach the question to an exact object

Pick the concept, equation, source, code witness, claim, misconception, or demo state before asking for help. The handoff stays grounded to that object.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptAlignment

Reward Hacking: Overoptimizing Preference Proxies

Anchored question

What is the smallest example that makes Reward Hacking: Overoptimizing Preference Proxies click without losing the math?

Source boundaryInspect source ids: amodei-2016-concrete-safety, gao-2022-reward-overoptimization, manheim-2018-goodhart, christiano-2017-human-preferencesStable content-object key attached

Role lenses for this object

These are fixed, deterministic perspectives derived from the selected object. They do not represent people, community contributions, or independent review.

Learner evidence requestAsk what would make "Reward Hacking: Overoptimizing Preference Proxies" feel predictable rather than familiar.

Assumption

Source ids amodei-2016-concrete-safety, gao-2022-reward-overoptimization, manheim-2018-goodhart, christiano-2017-human-preferences must support the exact object, not just the surrounding topic.

Source-checking summary

Treat this as a mechanism object: connect the definition to one equation, code witness, or demo before broadening the discussion.

Proposed experiment

Ask the learner to perturb one representation, then check whether the same invariant survives in math, code, and demo.

Next action

The learner can state the mechanism in their own words

Evidence4 checks

PredictionChecking carried observation

ActionReady for one action

AILearner handoff ready

Open source object

01PredictionChecking browser-local route memory

02EvidenceChecking for a carried observation

03BoundaryInspect source ids: amodei-2016-concrete-safety, gao-2022-reward-overoptimization, manheim-2018-goodhart, christiano-2017-human-preferences

04Next moveSave one next action

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays locally in this browser for concept:alignment/reward-hacking.

Draft noteNext action

No local draft saved.

Evidence to inspect

Source ids to inspect: amodei-2016-concrete-safety, gao-2022-reward-overoptimization, manheim-2018-goodhart, christiano-2017-human-preferences
Definition, prerequisite, and contrast concept links
The equation or code witness that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Object-attached AI handoff

I am working in Continuous Function's research reading room. Object: concept - Reward Hacking: Overoptimizing Preference Proxies Object key: concept:alignment/reward-hacking Context: Alignment Anchor id: concept/concept-notebook/alignment/reward-hacking Open question: What is the smallest example that makes Reward Hacking: Overoptimizing Preference Proxies click without losing the math? Evidence to inspect: - Source ids to inspect: amodei-2016-concrete-safety, gao-2022-reward-overoptimization, manheim-2018-goodhart, christiano-2017-human-preferences - Definition, prerequisite, and contrast concept links - The equation or code witness that makes the concept operational - One demo state that shows the invariant instead of a slogan Deterministic role lenses for this object: - Boundary: fixed perspectives, not people, community contributions, or independent review - Source-checking summary: Treat this as a mechanism object: connect the definition to one equation, code witness, or demo before broadening the discussion. - Proposed experiment: Ask the learner to perturb one representation, then check whether the same invariant survives in math, code, and demo. - Teach/transfer move: Turn the mechanism into one sentence that predicts a neighboring concept. - Assumptions: - Source ids amodei-2016-concrete-safety, gao-2022-reward-overoptimization, manheim-2018-goodhart, christiano-2017-human-preferences must support the exact object, not just the surrounding topic. - The stable content-object key lets local drafts, prompts, and route memory attach without changing the source page. - The concept explanation is local atlas prose until checked against its math, code, and source support. - Prerequisite gaps should become a repair route, not a reason to leave the object vague. - Role-lens requests: - Learner: ask for "Ask what would make "Reward Hacking: Overoptimizing Preference Proxies" feel predictable rather than familiar." | assumption: Source ids amodei-2016-concrete-safety, gao-2022-reward-overoptimization, manheim-2018-goodhart, christiano-2017-human-preferences must support the exact object, not just the surrounding topic. | next action: The learner can state the mechanism in their own words - Researcher: ask for "Source ids to inspect: amodei-2016-concrete-safety, gao-2022-reward-overoptimization, manheim-2018-goodhart, christiano-2017-human-preferences" | assumption: The stable content-object key lets local drafts, prompts, and route memory attach without changing the source page. | next action: The learner can name the prerequisite that would repair confusion - Experimenter: ask for "Choose one variable or condition to perturb before asking for an explanation." | assumption: The concept explanation is local atlas prose until checked against its math, code, and source support. | next action: The learner can predict how the mechanism changes under one perturbation - Professor: ask for "Find the smallest transferable rule a learner could reuse without the AI." | assumption: Prerequisite gaps should become a repair route, not a reason to leave the object vague. | next action: Teach or transfer: Turn the mechanism into one sentence that predicts a neighboring concept. What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action. Current deterministic role lens for this object: - Role lens: Learner - Evidence request: Ask what would make "Reward Hacking: Overoptimizing Preference Proxies" feel predictable rather than familiar. - Assumption to keep visible: Source ids amodei-2016-concrete-safety, gao-2022-reward-overoptimization, manheim-2018-goodhart, christiano-2017-human-preferences must support the exact object, not just the surrounding topic. - Proposed experiment: Ask the learner to perturb one representation, then check whether the same invariant survives in math, code, and demo. - Next action: The learner can state the mechanism in their own words

concept/concept-notebook/alignment/reward-hacking
concept:alignment/reward-hacking

Learning Map

Before / Now / Try / Next

BeforeRLHF: Reward Modeling + KL-Regularized Policy Optimization

NowIntuition → Math → Code → Demo

TryManipulate one control and predict the visible change.

NextProcess Reward Models: Step-Level Verifiers for Reasoning

Learnerpicture first Researchersource check Experimentertest the demo Professorname invariant

01Predict 02Ground 03Code 04Carry

01Intuitionready
02Mathready
03Codeready
04Interactive Demoready

Object Companion

Ask beside the selected object

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Alignment concept. Learning surface: Reward Hacking: Overoptimizing Preference Proxies. What this page says: When an imperfect preference proxy is optimized past its validation regime, policy mass shifts toward reward-model errors; KL, ensembles, LCBs, and monitoring slow this down but do not make the proxy true. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Copy-only tool: this panel composes a prompt for an AI tool you already use. Nothing you enter here is sent from this page.

Domain

Alignment

alignmentreward-hackinggoodhartoveroptimization

Prerequisites

RLHF: Reward Modeling + KL-Regularized Policy Optimization

Leads To

Process Reward Models: Step-Level Verifiers for Reasoning

Direct Preference Optimization Kahneman-Tversky Optimization Scaling Laws & Emergent Abilities

Within this domain