RLHF: Reward Modeling + KL-Regularized Policy Optimization

RLHF trains a reward model from pairwise preferences, then reweights a reference policy toward high learned reward while a KL penalty limits distribution shift.

published · difficulty 4/5 · 20 min read

Intuition

Build the mental picture first so the rest of the page has something to attach to.

PredictName the object in plain language, then predict what should change.Leave with one reusable mental picture before notation appears.

Canonical sources: Christiano et al., "Deep Reinforcement Learning from Human Preferences", and Ouyang et al., "Training language models to follow instructions with human feedback".

A pretrained or supervised-finetuned language model gives a distribution over completions. In InstructGPT-style RLHF, the reference policy is often the supervised-finetuned model trained from demonstrations. RLHF asks:

When humans prefer one completion over another, how should that preference move probability mass?

The mechanism has two stages.

First, train a reward model from comparisons. It does not learn an absolute moral score; it learns differences that make preferred completions more likely under a pairwise preference model.

Second, optimize a policy against that learned reward while penalizing movement away from a reference model. In the finite-action picture:

RLHF multiplies the reference probability of each completion by an exponential reward bonus, then renormalizes.

High reward pulls probability upward. The KL penalty controls how far the new policy may drift. If the reward model is a proxy with exploitable errors, optimizing too aggressively can move probability mass toward outputs that score well under the proxy but are worse under the real target.

Section prompt

Copy-only prompts — each action copies a page-grounded prompt to your clipboard. Nothing is sent by this site.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

InspectTrack the same object through the notation and check each symbol.Leave with the invariant the equations preserve.

Equation 1Inspect equation object\Delta_\phi(x,y_w,y_\ell) = r_\phi(x,y_w)-r_\phi(x,y_\ell), \qquad P_\phi(y_w\succ y_\ell\mid...Equation 2Inspect equation objectJ_x(\pi)= \sum_y \pi(y\mid x)r_\phi(x,y) - \betaKL(\pi(\cdot\mid x)\|\pi_{ref}(\cdot\mid x)),...Equation 3Inspect equation object\Delta = r_\phi(x,y_w)-r_\phi(x,y_\ell).

The two moving pieces are the preference-trained reward gap and the KL-shaped policy update:

\Delta_\phi(x,y_w,y_\ell) = r_\phi(x,y_w)-r_\phi(x,y_\ell), \qquad P_\phi(y_w\succ y_\ell\mid x)=\sigma(\Delta_\phi).

J_x(\pi)= \sum_y \pi(y\mid x)r_\phi(x,y) - \beta\mathrm{KL}(\pi(\cdot\mid x)\|\pi_{\mathrm{ref}}(\cdot\mid x)), \qquad \pi^*(y\mid x)= \frac{\pi_{\mathrm{ref}}(y\mid x)\exp(r_\phi(x,y)/\beta)} {\sum_{y'}\pi_{\mathrm{ref}}(y'\mid x)\exp(r_\phi(x,y')/\beta)}.

The rest of this section unpacks those two witnesses.

Preference data and reward-model likelihood

Let a preference datum be $(x,y_w,y_\ell)$ , where $y_w$ is preferred to $y_\ell$ for prompt $x$ . Define the reward gap

\Delta = r_\phi(x,y_w)-r_\phi(x,y_\ell).

A reward model $r_\phi(x,y)\in\mathbb R$ predicts

P_\phi(y_w\succ y_\ell\mid x)=\sigma(\Delta).

The negative log-likelihood is

\mathcal L_{\mathrm{RM}}(\phi) = -\frac{1}{N} \sum_{i=1}^N \log\sigma(\Delta_i),

where

\Delta_i = r_\phi(x_i,y_{w,i}) - r_\phi(x_i,y_{\ell,i}).

Only reward differences inside the same prompt are observed. Therefore $r_\phi(x,y)+c(x)$ gives the same pairwise probabilities as $r_\phi(x,y)$ .

Reward scale also needs a convention. With a fixed logistic noise model, reward gaps are measured in that model's log-odds units. With perfectly separable hard preferences, an unregularized reward model can drive gaps toward infinity. Practical systems therefore normalize, regularize, or otherwise choose a usable reward scale before policy optimization.

KL-regularized policy optimization

For a fixed prompt $x$ , reference policy $\pi_{\mathrm{ref}}$ , and candidate policy $\pi$ , define expected reward

R_x(\pi) = \sum_y \pi(y\mid x)r_\phi(x,y)

and the KL term

K_x(\pi)=\mathrm{KL}(\pi(\cdot\mid x)\|\pi_{\mathrm{ref}}(\cdot\mid x)).

The RLHF objective is

J_x(\pi)=R_x(\pi)-\beta K_x(\pi).

Equivalently, if

q_y = \log \frac{\pi(y\mid x)} {\pi_{\mathrm{ref}}(y\mid x)},

then

J_x(\pi) = \sum_y \pi(y\mid x)\{r_\phi(x,y)-\beta q_y\}.

Assume $\pi_{\mathrm{ref}}(y\mid x)>0$ on the candidate support. Optimize $J_x$ over the finite distribution $\pi(\cdot\mid x)$ subject to $\sum_y \pi(y\mid x)=1$ . The Lagrange stationarity condition is

r_\phi(x,y)-\beta(q_y+1)+\lambda=0.

Solving for $\pi(y\mid x)$ and normalizing gives

\pi^*(y\mid x) = \frac{ \pi_{\mathrm{ref}}(y\mid x)\exp(r_\phi(x,y)/\beta) }{ Z(x) }.

Here

Z(x) = \sum_{y'} \pi_{\mathrm{ref}}(y'\mid x) \exp(r_\phi(x,y')/\beta).

Larger $\beta$ keeps the policy closer to the reference. Smaller $\beta$ lets learned reward dominate.

Rearranging gives the bridge to DPO:

r_\phi(x,y) = \beta\log \frac{\pi^*(y\mid x)} {\pi_{\mathrm{ref}}(y\mid x)} + \beta\log Z(x).

The final term depends only on the prompt, so it cancels in preference differences.

PPO is the optimizer, not the definition

In language-model RLHF, the candidate space is enormous. InstructGPT-style RLHF uses PPO to optimize the learned reward with a KL penalty to the supervised-finetuned or reference model. The clean finite-action formula above is the exact optimum of a finite-action regularized objective. PPO is a stochastic optimizer for the language-model version; it is not this closed-form update, and a parametric policy with per-token KL details need not trace the toy optimum exactly.

Reward hacking

If $r_\phi(x,y)$ differs from the real target $u(x,y)$ , then low $\beta$ can concentrate the policy on outputs with high proxy reward and low true utility. KL regularization reduces this pressure but does not make the proxy correct.

Section prompt

Copy-only prompts — each action copies a page-grounded prompt to your clipboard. Nothing is sent by this site.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

TraceMatch variables to symbols before reading the implementation.Leave with a runnable witness for the math.

Code witness 1import numpy as np def sigmoid(z): return 1.0 / (1.0 + np.exp(-z)) def softmax(logits): logit...python

This witness keeps the pieces visible: pairwise reward-model fitting, reward-shift invariance, and the KL-regularized policy that reweights a reference distribution.

import numpy as np

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

def softmax(logits):
    logits = logits - logits.max()
    e = np.exp(logits)
    return e / e.sum()

# One prompt, four candidate completions.
# Shape: rewards, pi_ref, pi_star are all (K,).
pairs = np.array([
    [0, 1],
    [0, 2],
    [2, 1],
    [3, 1],
    [0, 3],
])

K = 4
r = np.zeros(K)
lr = 0.3
l2 = 0.05

for _ in range(800):
    grad = l2 * r

    for winner, loser in pairs:
        margin = r[winner] - r[loser]
        p = sigmoid(margin)
        g = p - 1.0
        grad[winner] += g
        grad[loser] -= g

    r -= lr * grad / len(pairs)
    r -= r.mean()  # choose one representative of the shift-equivalence class

pi_ref = np.array([0.35, 0.30, 0.20, 0.15])
beta = 0.7

def kl_regularized_policy(reward):
    logits = np.log(pi_ref) + reward / beta
    return softmax(logits)

pi_star = kl_regularized_policy(r)
pi_shifted = kl_regularized_policy(r + 10.0)

kl = np.sum(pi_star * (np.log(pi_star) - np.log(pi_ref)))

print("learned reward representative:", np.round(r, 3))
print("reference policy:             ", np.round(pi_ref, 3))
print("KL-regularized policy:        ", np.round(pi_star, 3))
print("KL(pi* || pi_ref):            ", round(float(kl), 4))

assert np.allclose(pi_star, pi_shifted)

Adding a constant to every reward changes neither pairwise preference probabilities nor the KL-regularized policy. The policy depends on reward differences and on how strongly $\beta$ anchors it to the reference.

Section prompt

Copy-only prompts — each action copies a page-grounded prompt to your clipboard. Nothing is sent by this site.

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

ManipulateChange one control and predict the visible response before reveal.Leave with the observed invariant or a repaired model.

Live Concept Demo

Explore RLHF: Reward Modeling + KL-Regularized Policy Optimization

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5undergraduatecode-aligned

Demo inquiry checkpoint

Manipulate one control and predict the visible change.

01Choose lensTrace a quantity

02ObserveDemo state pending

03GroundName the equation, invariant, or control that explains it.

04CarryNext: Direct Preference Optimization

Choose what to inspect in RLHF: Reward Modeling + KL-Regularized Policy Optimization. This shared fallback is an observation guide, not evidence of learning.

Loading interactive demo...

Use the demo as a probability-shaping machine. Change $\beta$ , toggle a proxy reward gap, and add a reward shift. Watch the reference policy get reweighted, and notice that shifting every reward leaves the policy unchanged.

Section prompt

Copy-only prompts — each action copies a page-grounded prompt to your clipboard. Nothing is sent by this site.

Object

4/4 sections ready

Concept: RLHF: Reward Modeling + KL-Regularized Policy Optimization

What is the smallest example that makes RLHF: Reward Modeling + KL-Regularized Policy Optimization click without losing the math?

BeforeMaximum LikelihoodNow4/4 sections readyTryManipulate one control and predict the visible change.NextDirect Preference Optimization

Object contextAlignment

StateAlignment

2 sources attachedChecking local snapshot

concept:alignment/rlhf

Math3 equation objects\Delta_\phi(x,y_w,y_\ell) = r_\phi(x,y_w)-r_\phi(x,y_\ell), \qquad P_\phi(y_w\succ y_\ell\mid...CodeRLHF: Reward Modeling + KL-Regularized Policy Optimization code witne...assert np.allclose(pi_star, pi_shifted)DemoRLHF: Reward Modeling + KL-Regularized Policy Optimization predictionManipulate one control and predict the visible change.

ConceptLearner lens

RLHF: Reward Modeling + KL-Regularized Policy Optimization

What is the smallest example that makes RLHF: Reward Modeling + KL-Regularized Policy Optimization click without losing the math?

Mode questionCan I say the mechanism back in one sentence before I reveal anything?

Start with the prediction checkpoint, then compare the reveal to the mental model.

Take this move

Study modes

Keep the object fixed; change the lens.

01PredictCommit before reveal.02ExplainName the invariant.03Ground2 source links nearby.04ExperimentChange one knob.05CarrySave the next question.

Route back through the notebook

Carry the same object through intuition, math, code, and demo.

4/4 sections ready

Carry inMaximum Likelihood

Bring the mental model from Maximum Likelihood; this page will reuse it instead of restarting from zero.

Work hereRLHF: Reward Modeling + KL-Regularized Policy Optimization

RLHF trains a reward model from pairwise preferences, then reweights a reference policy toward high learned reward while a KL penalty limits distribution shift.

Carry outDirect Preference Optimization

The next edge should feel earned: use the demo prediction here before following Direct Preference Optimization.

After The First Pass

Turn the concept into an inspected object.

The lower panels are one second act: keep the object fixed, inspect it visually, check source boundaries, practice transfer, then attach the research question.

ConceptRLHF: Reward Modeling + KL-Regularized Policy OptimizationAlignment

01MechanismInspect motionSee how the idea changes form before adding more explanation.02VisualChoose a cueBind one visible mark to one mathematical role.03SourcesCheck boundary2 source links attached to the object.04ClaimAudit evidenceTreat the page claim as something to verify.05PracticeAttempt transferWrite before feedback so the model has something to repair.06RoomAsk groundedAttach the question to the same exact object.

Mechanism Storyboard

See the idea move before the page explains it

RLHF trains a reward model from pairwise preferences, then reweights a reference policy toward high learned reward while a KL penalty limits distribution shift.

Demo notes open01 / Intuition

Editorial alignment illustration of human preference feedback shaping a reward landscape and updating a policy distribution.

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change RLHF: Reward Modeling + KL-Regularized Policy Optimization should make visible.

Visual Inquiry

Make the image answer a mathematical question

RLHF trains a reward model from pairwise preferences, then reweights a reference policy toward high learned reward while a KL penalty limits distribution shift.

4/4 stages readyDemo notes connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make RLHF: Reward Modeling + KL-Regularized Policy Optimization easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

Object - ConceptRLHF: Reward Modeling + KL-Regularized Policy OptimizationQuestion

What is the smallest example that makes RLHF: Reward Modeling + KL-Regularized Policy Optimization click without losing the math?

concept:alignment/rlhf

Boundary

sources: christiano-2017-human-preferences, ouyang-2022-instructgpt

Check

Open the closest source note before trusting the local explanation.

Evidence

2 selected-object sources shown first; 2 references total.

Next move

Audit the claim boundary, then ask from the same selected object.

selected object source · paper · 2017Deep reinforcement learning from human preferencesChristiano et al.

Located CF editorial boundary

Grounds preference-labeled reward models as a way to train behavior from comparative human feedback.

Used here as

Christiano trains a reward predictor from trajectory-segment comparisons and optimizes a policy on predicted reward. Ouyang describes demonstrations -> SFT, rankings -> reward model, and...

Caveat

Reviews only preference-modeling and KL-regularized optimization mechanics. It does not certify reward as true human objective, PPO exact attainment of the finite-action optimum, PPO-ptx/...

Open source

selected object source · paper · 2022Training language models to follow instructions with human feedbackOuyang et al.

Located CF editorial boundary

Grounds the modern instruction-following RLHF pipeline: demonstrations, preference rankings, reward model, and PPO.

Used here as

Christiano trains a reward predictor from trajectory-segment comparisons and optimizes a policy on predicted reward. Ouyang describes demonstrations -> SFT, rankings -> reward model, and...

Caveat

Reviews only preference-modeling and KL-regularized optimization mechanics. It does not certify reward as true human objective, PPO exact attainment of the finite-action optimum, PPO-ptx/...

Open source

Claim Review

RLHF trains a reward model from pairwise preferences, then reweights a reference policy toward high learned reward while a KL penalty limits distribution shift.

Object - ConceptRLHF: Reward Modeling + KL-Regularized Policy OptimizationQuestion

What is the smallest example that makes RLHF: Reward Modeling + KL-Regularized Policy Optimization click without losing the math?

concept:alignment/rlhf

Boundary

sources: christiano-2017-human-preferences, ouyang-2022-instructgpt

Check

Treat every claim as provisional until source support and a local witness agree.

Evidence

1 structured claim check on this concept.

Next move

Run the prediction or practice transfer before asking for a grounded review.

1 CF editorial source-scope review recorded

Publisher-side editorial review is not independent replication. Claims without it still need exact source-support review. 2 references and 3 local witnesses are available for inspection.

Preference-based RLHF learns a reward model from human comparisons and optimizes a policy against it; InstructGPT-style RLHF adds SFT and PPO with a KL penalty to the SFT/reference policy.

Registered CF editorial boundary

Deep reinforcement learning from human preferencesGrounds preference-labeled reward models as a way to train behavior from comparative human feedback.Training language models to follow instructions with human feedbackGrounds the modern instruction-following RLHF pipeline: demonstrations, preference rankings, reward model, and PPO.

Used here as

Christiano trains a reward predictor from trajectory-segment comparisons and optimizes a policy on predicted reward. Ouyang describes demonstrations -> SFT, rankings -> reward model, and PPO against RM rewar...

Local witness

source-span-christiano-2017-human-preferences source-span-ouyang-2022-instructgpt math-object-1 math-object-2 code-witness-1 interactive-demo

Equation 1

\Delta_\phi(x,y_w,y_\ell) = r_\phi(x,y_w)-r_\phi(x,y_\ell), \qquad P_\phi(y_w\succ y_\ell\mid x)=\sigma(\Delta_\phi).

Equation 2

J_x(\pi)= \sum_y \pi(y\mid x)r_\phi(x,y) - \beta\mathrm{KL}(\pi(\cdot\mid x)\|\pi_{\mathrm{ref}}(\cdot\mid x)), \qquad \pi^*(y\mid x)= \frac{\pi_{\mathrm{ref}}(y\mid x)\exp(r_\phi(x,y)/\beta)} {\sum_{y'}\pi_{\mathrm{ref}}(y'\mid x)\exp(r_\phi(x,y')/\beta)}.

Caveat

Reviews only preference-modeling and KL-regularized optimization mechanics. It does not certify reward as true human objective, PPO exact attainment of the finite-action optimum, PPO-ptx/pretraining-gradient...

Review stateCF editorial source-scope reviewClaim metadata: source checkedPublisher-side editorial review only; not independent replication. Check caveats and exact source scope.

Christiano supports learning a reward predictor from pairwise trajectory preferences and optimizing a policy on predicted reward. Ouyang supports InstructGPT demonstrations -> SFT, rankings -> RM, and PPO against RM with per-token KL to SFT. Local math/code/demo are toy witnesses for sigmoid preferences, KL probability shaping, shift invariance, and proxy-gap caveats.

Reviewer: codex+oracle; reviewed 2026-05-07

Practice notebook

Use the idea, then test it somewhere new

RLHF trains a reward model from pairwise preferences, then reweights a reference policy toward high learned reward while a KL penalty limits distribution shift.

AttemptNo learning claim inferred

Object - ConceptRLHF: Reward Modeling + KL-Regularized Policy OptimizationQuestion

What is the smallest example that makes RLHF: Reward Modeling + KL-Regularized Policy Optimization click without losing the math?

concept:alignment/rlhf

Boundary

sources: christiano-2017-human-preferences, ouyang-2022-instructgpt

Check

Use one state from RLHF: Reward Modeling + KL-Regularized Policy Optimization to explain what changes, why it changes, and which assumption the explanation needs.

Evidence

No learner move yet; no learning state is inferred.

Next move

Write first, use only the help you need, then try a new case without it.

Explain

Use one state from RLHF: Reward Modeling + KL-Regularized Policy Optimization to explain what changes, why it changes, and which assumption the explanation needs.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Attempt context

Learner move

Write an attempt before asking the companion.

Assistance used

0 of 3 progressive hints opened.

Evidence boundary

This draft and any AI response do not establish mastery; a later unassisted case can.

Misconception check

Grounded object roomClose

Selected object routeAsk from this object; carry one invariant back.sources: christiano-2017-human-preferences, ouyang-2022-instructgpt

ObjectConceptRLHF: Reward Modeling + KL-Regularized Policy Optimization
PredictBefore revealRLHF: Reward Modeling + KL-Regularized Policy Optimization prediction
WitnessCompare codeRLHF: Reward Modeling + KL-Regularized Policy Optimization code witne...
RoomAsk groundedChecking local snapshot

ConceptRLHF: Reward Modeling + KL-Regularized Policy OptimizationAlignment

Code witness comparisonRLHF: Reward Modeling + KL-Regularized Policy Optimization code witness 1assert np.allclose(pi_star, pi_shifted)Prediction before revealRLHF: Reward Modeling + KL-Regularized Policy Optimization predictionManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes RLHF: Reward Modeling + KL-Regularized Policy Optimization click without losing the math?Checking local snapshot

Research Room

Attach the question to an exact object

Pick the concept, equation, source, code witness, claim, misconception, or demo state before asking for help. The handoff stays grounded to that object.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptAlignment

RLHF: Reward Modeling + KL-Regularized Policy Optimization

Anchored question

What is the smallest example that makes RLHF: Reward Modeling + KL-Regularized Policy Optimization click without losing the math?

Source boundaryInspect source ids: christiano-2017-human-preferences, ouyang-2022-instructgptStable content-object key attached

Role lenses for this object

These are fixed, deterministic perspectives derived from the selected object. They do not represent people, community contributions, or independent review.

Learner evidence requestAsk what would make "RLHF: Reward Modeling + KL-Regularized Policy Optimization" feel predictable rather than familiar.

Assumption

Source ids christiano-2017-human-preferences, ouyang-2022-instructgpt must support the exact object, not just the surrounding topic.

Source-checking summary

Treat this as a mechanism object: connect the definition to one equation, code witness, or demo before broadening the discussion.

Proposed experiment

Ask the learner to perturb one representation, then check whether the same invariant survives in math, code, and demo.

Next action

The learner can state the mechanism in their own words

Evidence4 checks

PredictionChecking carried observation

ActionReady for one action

AILearner handoff ready

Open source object

01PredictionChecking browser-local route memory

02EvidenceChecking for a carried observation

03BoundaryInspect source ids: christiano-2017-human-preferences, ouyang-2022-instructgpt

04Next moveSave one next action

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays locally in this browser for concept:alignment/rlhf.

Draft noteNext action

No local draft saved.

Evidence to inspect

Source ids to inspect: christiano-2017-human-preferences, ouyang-2022-instructgpt
Definition, prerequisite, and contrast concept links
The equation or code witness that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Object-attached AI handoff

I am working in Continuous Function's research reading room. Object: concept - RLHF: Reward Modeling + KL-Regularized Policy Optimization Object key: concept:alignment/rlhf Context: Alignment Anchor id: concept/concept-notebook/alignment/rlhf Open question: What is the smallest example that makes RLHF: Reward Modeling + KL-Regularized Policy Optimization click without losing the math? Evidence to inspect: - Source ids to inspect: christiano-2017-human-preferences, ouyang-2022-instructgpt - Definition, prerequisite, and contrast concept links - The equation or code witness that makes the concept operational - One demo state that shows the invariant instead of a slogan Deterministic role lenses for this object: - Boundary: fixed perspectives, not people, community contributions, or independent review - Source-checking summary: Treat this as a mechanism object: connect the definition to one equation, code witness, or demo before broadening the discussion. - Proposed experiment: Ask the learner to perturb one representation, then check whether the same invariant survives in math, code, and demo. - Teach/transfer move: Turn the mechanism into one sentence that predicts a neighboring concept. - Assumptions: - Source ids christiano-2017-human-preferences, ouyang-2022-instructgpt must support the exact object, not just the surrounding topic. - The stable content-object key lets local drafts, prompts, and route memory attach without changing the source page. - The concept explanation is local atlas prose until checked against its math, code, and source support. - Prerequisite gaps should become a repair route, not a reason to leave the object vague. - Role-lens requests: - Learner: ask for "Ask what would make "RLHF: Reward Modeling + KL-Regularized Policy Optimization" feel predictable rather than familiar." | assumption: Source ids christiano-2017-human-preferences, ouyang-2022-instructgpt must support the exact object, not just the surrounding topic. | next action: The learner can state the mechanism in their own words - Researcher: ask for "Source ids to inspect: christiano-2017-human-preferences, ouyang-2022-instructgpt" | assumption: The stable content-object key lets local drafts, prompts, and route memory attach without changing the source page. | next action: The learner can name the prerequisite that would repair confusion - Experimenter: ask for "Choose one variable or condition to perturb before asking for an explanation." | assumption: The concept explanation is local atlas prose until checked against its math, code, and source support. | next action: The learner can predict how the mechanism changes under one perturbation - Professor: ask for "Find the smallest transferable rule a learner could reuse without the AI." | assumption: Prerequisite gaps should become a repair route, not a reason to leave the object vague. | next action: Teach or transfer: Turn the mechanism into one sentence that predicts a neighboring concept. What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action. Current deterministic role lens for this object: - Role lens: Learner - Evidence request: Ask what would make "RLHF: Reward Modeling + KL-Regularized Policy Optimization" feel predictable rather than familiar. - Assumption to keep visible: Source ids christiano-2017-human-preferences, ouyang-2022-instructgpt must support the exact object, not just the surrounding topic. - Proposed experiment: Ask the learner to perturb one representation, then check whether the same invariant survives in math, code, and demo. - Next action: The learner can state the mechanism in their own words

concept/concept-notebook/alignment/rlhf
concept:alignment/rlhf

Learning Map

Before / Now / Try / Next

BeforeMaximum Likelihood

NowIntuition → Math → Code → Demo

TryManipulate one control and predict the visible change.

NextDirect Preference Optimization

Learnerpicture first Researchersource check Experimentertest the demo Professorname invariant

01Predict 02Ground 03Code 04Carry

01Intuitionready
02Mathready
03Codeready
04Interactive Demoready

Object Companion

Ask beside the selected object

RLHF trains a reward model from pairwise preferences, then reweights a reference policy toward high learned reward while a KL penalty limits distribution shift.

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Alignment concept. Learning surface: RLHF: Reward Modeling + KL-Regularized Policy Optimization. What this page says: RLHF trains a reward model from pairwise preferences, then reweights a reference policy toward high learned reward while a KL penalty limits distribution shift. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Copy-only tool: this panel composes a prompt for an AI tool you already use. Nothing you enter here is sent from this page.

Domain

Alignment

alignmentrlhfreward-modelingpreferencesbradley-terrykl-regularization

Prerequisites

Maximum Likelihood Cross-Entropy KL Divergence (Relative Entropy)

Leads To

Direct Preference Optimization Reward Hacking: Overoptimizing Preference Proxies Process Reward Models: Step-Level Verifiers for Reasoning

Kahneman-Tversky Optimization

Within this domain