Test-Time Compute & Inference Scaling

Why It Matters for Modern Models

Test-time compute is the computation spent producing an answer at inference, for example by generating candidate answers or longer reasoning sequences. OpenAI reported that the performance of o1, a model trained to reason at length before answering, improves with more reinforcement-learning training and with more time spent reasoning at test time (OpenAI, 2024).
The best use of test-time compute depends on the question: Snell et al. (2025) found that allocating it per prompt according to difficulty was more than 4× as efficient as best-of-N, and that on problems where a smaller model already had some success, extra inference compute could outperform a 14× larger model at equal FLOPs.
A verifier makes search possible: Cobbe et al. (2021) generated many solutions to grade-school math problems and kept the one their trained verifier ranked highest, which improved accuracy significantly.

Questions to Think About

Best-of-N is only as good as its verifier; what happens when the verifier can be fooled, so that drawing more samples finds more answers that score well but are wrong?
How does scoring partial solutions with a process reward model let a search abandon a poor line of reasoning early, and what does that cost?

Interactive Demo

The interactive demo for this concept is in the newer notebook.

Open the notebook demo

Core Math

If you prefer intuition first, try the demo in the newer notebook, then come back for the math.

Key Equation

y^* = \arg\max_{y \in \{y_1, \dots, y_N\}} V(x, y)

Test-time compute is computation spent on each question at inference, for example by generating many candidate answers or longer chains of reasoning and choosing among them.

Best-of- $N$ sampling draws $N$ responses to a prompt $x$ and keeps the one a verifier $V$ scores highest (Cobbe et al., 2021):

y^* = \arg\max_{y \in \{y_1, \dots, y_N\}} V(x, y)

Process reward models score each intermediate step of a solution. Lightman et al. (2024) define the solution score as the probability, under the PRM, that every step is correct, implemented as the product of per-step predictions with neutral labels counted as positive:

R_{\text{PRM}}(s_1, \dots, s_k) = \prod_{i=1}^k P(\text{step } i \text{ correct} \mid s_1, \dots, s_i)

Tree search over reasoning steps decides which partial solution to extend next, for example with the upper-confidence rule of Monte Carlo tree search, where $V(s)$ is the estimated value of node $s$ and $N$ counts visits:

\text{UCB}(s) = V(s) + c\sqrt{\frac{\log N_{\text{parent}}}{N_s}}

How much accuracy improves with more inference compute is an empirical question that depends on the model, the task and how the compute is spent; it is not a fixed law (Snell et al., 2025).