Process Reward Models: Step-Level Verifiers for Reasoning

A process reward model scores intermediate reasoning steps rather than only the final answer. These learned scores can guide reranking and search, but remain proxies for correctness.

Advanced · Undergraduate mathematics · about 15 minutes

Reading map and next steps

Intuition

The idea in plain words, before the symbols.

A process reward model scores each intermediate reasoning step, while an outcome reward model gives one score for the completed solution. We compare complete reasoning traces that differ in their final-answer correctness and local step validity. The figure checks how outcome scoring, process scoring, and verifier errors change which trace receives the most probability. You will need the sigmoid, cross-entropy, and softmax over a finite set of choices.

A process reward model asks a local question at every step: given the prompt and the reasoning prefix so far, is this next step valid?

That changes the shape of feedback. Instead of treating a whole solution as one indivisible completion, the verifier sees a path through intermediate states. A lucky final answer with broken reasoning can receive high outcome reward and low process reward. A trace with one early algebra mistake can be rejected before the mistake compounds.

Uesato et al. (2022) measured this on grade-school math word problems: outcome supervision matched process supervision on final-answer error, but reducing reasoning errors among solutions whose final answer was correct required process supervision, or a learned reward model that imitates it.

The important caveat is that a process reward model is still a learned proxy. It can localize errors better than terminal-only feedback, but its step labels can be ambiguous, expensive, domain-limited, or wrong. PRMs do not solve reward hacking; they move the proxy from the leaf of the reasoning tree onto the edges.

Sources. Cobbe et al. (2021), "Training Verifiers to Solve Math Word Problems", train outcome verifiers that rerank sampled solutions. Uesato et al. (2022), "Solving math word problems with process- and outcome-based feedback", compare outcome and process supervision. Lightman et al. (2024), "Let's Verify Step by Step", train process reward models on the PRM800K step labels. Snell et al. (2025), "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters", use process reward models to guide search at inference time.

Study prompts for this section: Intuition

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Mathematics

Definitions, assumptions and the derivation.

Equation 1 Equation 2 Equation 3

Fix one prompt $x$ . In the simplified binary notation used here, a process reward model sees a reasoning prefix and a candidate next step, then predicts whether that step is locally correct. Lightman et al.'s PRM800K labels include positive, negative, and neutral ratings; here we collapse them into correct vs. not correct to show how the verifier works.

h=(a_1,\dots,a_t),\qquad c(h,a)\in\{0,1\},\qquad P_\phi(c=1\mid h,a)=\sigma(r_\phi(h,a)).

For labeled steps $(h_i,a_i,c_i)$ , the binary verifier can be trained with cross-entropy:

p_i=P_\phi(c=1\mid h_i,a_i),\qquad \mathcal L_{\mathrm{PRM}}(\phi) = - \sum_i \left[ c_i\log p_i + (1-c_i)\log(1-p_i) \right].

A complete solution is a terminal trace

\tau=(a_1,\dots,a_T).

Outcome supervision observes only a terminal label

z(\tau)\in\{0,1\}.

For a terminal trace, one simple additive score is the sum of step logits:

S_\phi(\tau) = \sum_{t=1}^{T} r_\phi(h_{t-1},a_t).

Other aggregations, such as mean step probability, product of step probabilities, or minimum step score, are design choices rather than a universal PRM definition. Lightman et al. define the solution score as the probability, under the PRM, that every step is correct, implemented as the product of per-step predictions with neutral labels counted as positive. The key definition is local verification of steps.

Separate from Lightman et al.'s best-of-N evaluation, this page also shows a finite KL-style RLHF bridge. A KL-regularized trajectory update is

\pi_\beta(\tau\mid x) = \frac{ \pi_{\mathrm{ref}}(\tau\mid x)\exp(S_\phi(\tau)/\beta) }{ \sum_{\tau'} \pi_{\mathrm{ref}}(\tau'\mid x)\exp(S_\phi(\tau')/\beta) }.

This is the same finite-action probability-shaping pattern as RLHF, but the learned reward decomposes over reasoning steps. Smaller $\beta$ pushes harder on the verifier score. If the step verifier is wrong, optimization can still select verifier error.

Study prompts for this section: Mathematics

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Code

The same calculation as runnable code, in the notation of the derivation.

Code example 1import math import numpy as np def logit(p): p = min(max(p, 1e-6), 1 - 1e-6) return math.log(...python

This code uses four fixed traces for the equation $2(x+3)=14$ . The outcome verifier prefers a trace with the correct final answer but invalid reasoning. The accurate process verifier prefers the clean trace. If we inject a false-positive process error, the process verifier can be fooled too.

import math
import numpy as np

def logit(p):
    p = min(max(p, 1e-6), 1 - 1e-6)
    return math.log(p / (1 - p))

def policy(scores, ref, beta=0.7):
    logits = np.log(ref) + np.asarray(scores) / beta
    logits = logits - logits.max()
    weights = np.exp(logits)
    return weights / weights.sum()

traces = ["clean", "lucky_final", "slip", "wrong"]
ref = np.array([0.30, 0.22, 0.28, 0.20])
outcome_p = np.array([0.86, 0.92, 0.16, 0.12])
step_p = [
    [0.93, 0.91],
    [0.16, 0.24],
    [0.93, 0.22],
    [0.20, 0.35],
]

def process_scores(step_probs):
    return [sum(logit(p) for p in probs) for probs in step_probs]

outcome_winner = traces[int(policy([logit(p) for p in outcome_p], ref).argmax())]
process_winner = traces[int(policy(process_scores(step_p), ref).argmax())]

hacked_steps = [list(probs) for probs in step_p]
hacked_steps[1] = [0.98, 0.97]
hacked_winner = traces[int(policy(process_scores(hacked_steps), ref).argmax())]

print(outcome_winner, process_winner, hacked_winner)
assert outcome_winner == "lucky_final"
assert process_winner == "clean"
assert hacked_winner == "lucky_final"

Study prompts for this section: Code

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Demo

Change an input, predict the result, and see which quantities respond.

ExploreChange one control at a time and name what responds.An optional observation guide follows the demo.

Explore Process Reward Models: Step-Level Verifiers for Reasoning

Loading the interactive demo…

Observation guide

Optional and ungraded. Opening a guide saves nothing.

Choose what to inspect in Process Reward Models: Step-Level Verifiers for Reasoning, then open its guide.

In the demo, switch between outcome-only scoring and process scoring, change the process aggregation, inject verifier errors, and lower $\beta$ to increase optimization pressure. Watch how probability mass moves across complete traces, and whether the selected trace has a correct final answer, valid local steps, or merely a high verifier score.

Study prompts for this section: Demo

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Focus on

In focus

Concept: Process Reward Models: Step-Level Verifiers for Reasoning

What is the smallest example of Process Reward Models you can work through by hand, and what does it show?

BeforeRLHF: Reward Modeling + KL-Regularized Policy OptimizationNextTree Search Reasoning: Allocating Inference Budget Across Prefixes

DetailsAlignment

About this itemAlignment

4 sources citedChecking this browser

Reference id

concept:alignment/process-reward-models

Math3 equationsh=(a_1,\dots,a_t),\qquad c(h,a)\in\{0,1\},\qquad P_\phi(c=1\mid h,a)=\sigma(r_\phi(h,a)).CodeProcess Reward Models: Step-Level Verifiers for Reasoning code example 1assert outcome_winner == "lucky_final"DemoProcess Reward Models: Step-Level Verifiers for Reasoning interactive...Change one control at a time and name what responds.

After the first pass

Go deeper

Optional material for a second reading. Sources are always shown; the other panels open from their headings or from these links, and closing one keeps what you wrote or revealed.

What to look for in each sectionChoose a section and what you expect to see in it, then read short advice.

Section by section

Choose what to look for in each section

A process reward model scores intermediate reasoning steps rather than only the final answer. These learned scores can guide reranking and search, but remain proxies for correctness.

Choose what to look for (no demo yet)01 / Intuition

Illustration of a branching reasoning trace with step-level verifier scores and a fading invalid path.

What to look for

Start with the picture, metaphor, or geometric mechanism.

Choose first

Choose what you expect to see in this section of Process Reward Models: Step-Level Verifiers for Reasoning.

Questions to ask of the pictureChoose a part of the picture and what you expect, then read short advice.

Looking at the picture

Questions to ask of the picture

A process reward model scores intermediate reasoning steps rather than only the final answer. These learned scores can guide reranking and search, but remain proxies for correctness.

4 of 4 sections writtenNo live demo yet

Visual cueWhich part of the picture should you look at first?

Highlight size2/4

Question

Which part of the picture should you look at first?

Choose first

Pick the part of the picture you expect to explain Process Reward Models: Step-Level Verifiers for Reasoning.

Sources

References for this page

A reference shows where an idea comes from; it does not vouch for every step on this page.

paper · 2024Let's Verify Step by StepLightman et al.

Why it is cited

ICLR 2024. Trains process reward models on PRM800K, human labels for each step of model-written solutions to MATH problems, and finds they select correct solutions more reliably than outcome reward models in best-of-N search.

Open source

paper · 2022Solving math word problems with process- and outcome-based feedbackUesato et al.

Why it is cited

arXiv preprint. On GSM8K, outcome supervision matched process supervision on final-answer error, but process supervision, or a reward model that imitates it, was needed to cut reasoning errors among solutions with correct final answers.

Open source

paper · 2021Training Verifiers to Solve Math Word ProblemsCobbe et al.

Why it is cited

arXiv preprint. Introduces GSM8K and trains outcome verifiers that judge whole solutions by their final answer, then reranks many sampled solutions with them.

Open source

paper · 2025Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model ParametersSnell et al.

Why it is cited

ICLR 2025, under a shorter title. Uses process reward models as verifiers for best-of-N and beam search at inference time, and finds that the best search strategy depends on how hard the prompt is.

Open source

What each claim rests on1 claim, each with its sources, where it appears on this page, and its caveats.

Claims and sources

A process reward model scores intermediate reasoning steps rather than only the final answer. These learned scores can guide reranking and search, but remain proxies for correctness.

Checked against the cited passages

The checks were made by this site, not by an independent reviewer.

Process reward models are step-level verifiers rather than only final-answer judges: for prefix h and next step a, a binary formulation estimates P_phi(c=1|h,a)=sigma(r_phi(h,a)), enabling reranking or search while remaining a learned proxy.

Sources

Let's Verify Step by StepICLR 2024. Trains process reward models on PRM800K, human labels for each step of model-written solutions to MATH problems, and finds they select c...

What the sources support

Lightman et al. contrast outcome reward models, trained on final results, with process reward models trained on a label for each step; they combine the step scores to rank whole solutions in best-of-N search.

Where to see it on this page

Let's Verify Step by Step Equation 1 Equation 2 Code example 1 Demo

Equation 1

h=(a_1,\dots,a_t),\qquad c(h,a)\in\{0,1\},\qquad P_\phi(c=1\mid h,a)=\sigma(r_\phi(h,a)).

Equation 2

p_i=P_\phi(c=1\mid h_i,a_i),\qquad \mathcal L_{\mathrm{PRM}}(\phi) = - \sum_i \left[ c_i\log p_i + (1-c_i)\log(1-p_i) \right].

Caveat

The binary label and cross-entropy loss simplify Lightman et al.'s positive, negative and neutral labels. Not covered: the best way to combine step scores, calibration, training a generator against a PRM, ga...

StatusChecked against the cited passages.

Lightman et al. train process reward models on step-level labels and evaluate them by best-of-N selection of the highest-ranked generated solution. The binary cross-entropy notation here simplifies their positive, negative and neutral labels. Their examples of reward-model mistakes and false positives support the point that a PRM is still a proxy. The KL reweighting on this page is not part of the checked claim.

Explain it without the pageWrite your own explanation before asking for feedback. Your draft and any hints stay while this page is open.

Practice · Process Reward Models: Step-Level Verifiers for Reasoning

Try the idea in your own words

A process reward model scores intermediate reasoning steps rather than only the final answer. These learned scores can guide reranking and search, but remain proxies for correctness.

Concept · Selected for practice

Process Reward Models: Step-Level Verifiers for Reasoning

What it rests on: Sources: Let's Verify Step by Step; Solving math word problems with process- and outcome-based feedback; Training Verifiers to Solve Math Word Prob...

Context and links

Alignment

Choose a task

Explain the mechanism

For Process Reward Models: Step-Level Verifiers for Reasoning: What is the smallest example of Process Reward Models you can work through by hand, and what does it show? Explain your answer, including what changes, why, and which assumption matters.

Your answerNo answer yet

A rough first thought is enough. Your draft stays when you change tasks.

Your draft stays on this page and is cleared when you leave or reload.

A little help · Explain

Open one hint at a time. These are suggestions, not your answer or a grade.

0 of 3 hints shown for this question.

Clearing your answer keeps this help record. Outside help is your own report; this page cannot check it.

Where am I stuck? (optional)

Your own description, not an automatic diagnosis

Choose one, or leave this unspecified. Select it again to clear it.

Take your draft to a feedback conversation

No AI feedback runs here. You can copy a prompt to use elsewhere; nothing is sent automatically. Review the text before sharing, and leave out private information.

Write an attempt before copying a feedback prompt.

A draft, or an AI reply to it, is not a test of what you have learned. To check that, try a different case later without help.

Ask about this pagePick one item on this page and copy a prompt about it into an AI assistant you already use. Notes you write here stay in this browser.

Ask about this pageClose

ConceptProcess Reward Models: Step-Level Verifiers for ReasoningSources: Let's Verify Step by Step; Solving math word problems with process- and outcome-based feedback; Training Verifiers to Solve Math Word Prob...assert outcome_winner == "lucky_final"

Your question

Choose what your question is about

Pick the idea, equation, source, code, claim, misconception or demo state you want to ask about. The prompt you copy includes it, so the answer can stay on that item.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptAlignment

Process Reward Models: Step-Level Verifiers for Reasoning

Starting question

What is the smallest example of Process Reward Models you can work through by hand, and what does it show?

SourcesCheck the 4 cited sources listed under Sources.Notes can be saved for this item

Points of view on this item

Fixed prompts generated from this item. They are not comments from people or independent reviews.

Learner’s first stepAsk what would make "Process Reward Models: Step-Level Verifiers for Reasoning" feel predictable rather than familiar.

Assumption

The 4 cited sources must support this exact item, not just the surrounding topic.

Source-checking summary

Connect the definition to one equation, piece of code or demo before widening the discussion.

Proposed experiment

Change one input, then check that the same relationship holds in the mathematics, the code and the demo.

Next action

You can state the mechanism in your own words

Evidence4 checks

ObservationChecking for a saved observation

ActionReady for one action

PromptLearner prompt ready to copy

Go to this item

01ObservationChecking what this browser has saved

02EvidenceChecking for a saved observation

03SourcesCheck the 4 cited sources listed under Sources.

04Next stepSave one next action

Local action draftNo local draft saved yetOpen when you are ready to write one next action

Local action draft

This draft stays in this browser and is attached to this item only.

Draft noteNext action

No local draft saved.

Evidence to inspect

What the 4 cited sources say about this exact item
The definition, its prerequisites, and a contrasting concept
The equation or code that makes the concept concrete
One demo state that shows what stays the same as the inputs change

What would resolve this

You can state the mechanism in your own words
You can name the prerequisite that would clear up the confusion
You can predict how the result changes when one input changes

Reference id

concept/concept-notebook/alignment/process-reward-models
concept:alignment/process-reward-models

Return to reading

Reading map

In this concept

Change one control at a time and name what responds.

Try the demo

Continue the curriculum

Part VII · unit 5 of 7

Tree Search Reasoning: Allocating Inference Budget Across Prefixes

Connections and sources

Domain

Alignment

alignment
process-supervision
reward-modeling
reasoning
verifiers

Prerequisites

Leads to

Study aid

Ask an assistant about this notebook

Compose a question with this notebook as context, then copy it into an AI assistant you already use.

Compose a question

Your question

GoalComfortStyleStuck on

Copy-only tool: this panel composes a prompt from this page and your choices above, for an AI assistant you already use. Nothing you enter here is sent from this page.