Continuous Function

Foundation reference

Process Reward Models

Step-level feedback says where a solution went wrong, not only that it did. Uesato et al. (2022) found that supervising only final answers gave similar final-answer error rates on grade-school math problems, but that step-level supervision, or a reward model that imitates it, was needed for the reasoning itself to be correct.

A newer notebook covers this concept. This older page keeps its summary, math and sources.

Concept 81 of 100Scaling & AlignmentStudy phase 11
Open the notebook demoOpen the newer notebookAll concepts
Key equation
RPRM(y)=∏k=1KP(sk correct∣s1,…,sk)R_{\text{PRM}}(y) = \prod_{k=1}^{K} P(s_k \text{ correct} \mid s_1, \dots, s_k)
Reading map and next steps

Study guide

Where to start

Concept 81 of 100Process Reward ModelsScaling & Alignment · Study phase 11: Reasoning at inference time & synthetic data
Question

An outcome label gives one bit for a whole solution; why does a label for each step make it easier to learn which step caused an error?

Open the notebook demoGo to the mathOpen the newer notebookNext concept
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.

Before you use the demo in the newer notebook, say what you expect to change.

CheckCheck against the math and the papers.

Compare what the notebook demo shows with the key equation here and with the papers cited below.

GeneralizeName what stays the same.

Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.

ContinueContinue in the newer notebook.

The newer notebook covers the same concept and has an interactive demo.

Why It Matters for Modern Models

  • Step-level feedback says where a solution went wrong, not only that it did. Uesato et al. (2022) found that supervising only final answers gave similar final-answer error rates on grade-school math problems, but that step-level supervision, or a reward model that imitates it, was needed for the reasoning itself to be correct.
  • Trained on 800,000 human step-level labels (the PRM800K dataset), a process reward model used to choose among many sampled solutions solved 78% of a representative subset of the MATH test set, outperforming an outcome reward model used the same way (Lightman et al., 2024).
  • Step labels are expensive to collect by hand. Math-Shepherd labels a step automatically by generating several completions from it and checking whether any of them reach the correct final answer (Wang et al., 2024).

Questions to Think About

  • A solution can contain a wrong step and still reach the right answer; why does that make outcome-only supervision reward flawed reasoning?
  • When a reward model chooses among N samples, why does verifier quality matter more as N grows, and what happens when the policy finds steps the verifier wrongly approves?

Interactive Demo

The interactive demo for this concept is in the newer notebook.

Open the notebook demo

Core Math

If you prefer intuition first, try the demo in the newer notebook, then come back for the math.

Key Equation
RPRM(y)=∏k=1KP(sk correct∣s1,…,sk)R_{\text{PRM}}(y) = \prod_{k=1}^{K} P(s_k \text{ correct} \mid s_1, \dots, s_k)

A multi-step solution yy is a sequence of steps s1,…,sKs_1, \dots, s_K. Two kinds of reward model can score it.

An outcome reward model (ORM) is trained on whether final answers are correct, and predicts whether the whole solution is right:

RORM(y)=P(final answer correct∣y)R_{\text{ORM}}(y) = P(\text{final answer correct} \mid y)

A process reward model (PRM) is trained on labels for individual steps, and predicts for each step whether it is correct given the steps before it, pk=P(sk correct∣s1,…,sk)p_k = P(s_k \text{ correct} \mid s_1, \dots, s_k). Lightman et al. (2024) define the solution score as the probability, under the PRM, that every step is correct, implemented as the product of per-step predictions with neutral labels counted as positive:

RPRM(y)=∏k=1KpkR_{\text{PRM}}(y) = \prod_{k=1}^{K} p_k

Math-Shepherd uses the minimum over steps instead (Wang et al., 2024); either way, a solution is judged by its weakest steps.

Best-of-N uses a reward model as a verifier: sample NN candidate solutions and return the highest-scoring one,

y∗=arg⁡max⁡i≤NR(yi).y^* = \arg\max_{i \le N} R(y_i).

Key Papers

Let's Verify Step by Step

Lightman et al.2024ICLR
Read paper →

Solving math word problems with process- and outcome-based feedback

Uesato et al.2022arXiv
Read paper →

Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations

Wang et al.2024ACL
Read paper →

Connections

Prerequisites

RLHFTest-Time Compute

Where to go next

A newer notebook covers Process Reward Models, with an interactive demo.

Continue in the newer notebook