Continuous Function

Foundation reference

Model-Graded Evaluations

A model-graded evaluation uses a judge model to score responses against a rubric or to compare pairs of responses. Zheng et al. (2023) found that strong judges such as GPT-4 agreed with human preferences over 80% of the time on MT-Bench and Chatbot Arena comparisons, about as often as people agreed with each other.

Concept 92 of 100Scaling & AlignmentStudy phase 12
Go to the mathAll concepts
Key equation
μ^=1n∑i=1nE(xi,yi;R)\hat{\mu} = \frac{1}{n} \sum_{i=1}^{n} E(x_i, y_i; R)
Reading map and next steps

Study guide

Where to start

Concept 92 of 100LLM-as-JudgeScaling & Alignment · Study phase 12: Advanced alignment & safety research
Question

If a judge agrees with people on 80% of comparisons, how far can it be trusted to rank two models that differ only slightly?

Go to the mathAll conceptsNext concept
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.

Pick a concrete input and say what the equation should give before you read the math.

CheckCheck against the math and the papers.

Work through the math, then read the cited papers for the original argument.

GeneralizeName what stays the same.

Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.

ContinueFollow the connections.

Use the prerequisites and related concepts at the end of the page to fill the next gap.

Why It Matters for Modern Models

  • A model-graded evaluation uses a judge model to score responses against a rubric or to compare pairs of responses. Zheng et al. (2023) found that strong judges such as GPT-4 agreed with human preferences over 80% of the time on MT-Bench and Chatbot Arena comparisons, about as often as people agreed with each other.
  • Benchmarks such as MT-Bench and AlpacaEval use a model as the judge (Zheng et al., 2023; Dubois et al., 2024). Chatbot Arena, by contrast, ranks models from crowdsourced human votes between two anonymous responses (Chiang et al., 2024).
  • Judges have systematic biases, including preferences for answers in a particular position, for longer answers and for their own outputs (Zheng et al., 2023). Controlling for length raised the correlation of AlpacaEval rankings with Chatbot Arena rankings from 0.94 to 0.98 (Dubois et al., 2024).

Questions to Think About

  • Why does asking twice with the two responses swapped reduce position bias, and what does it cost?
  • What goes wrong when a model is trained or selected to score well with the same judge that is used to evaluate it?

Core Math

Key Equation
μ^=1n∑i=1nE(xi,yi;R)\hat{\mu} = \frac{1}{n} \sum_{i=1}^{n} E(x_i, y_i; R)

In a model-graded evaluation a judge model EE scores outputs against a rubric RR, either one response at a time,

si=E(xi,yi;R),s_i = E(x_i, y_i; R),

or by comparing two responses to the same prompt and saying which is better. Scores over a test set of nn prompts are averaged,

μ^=1n∑i=1nsi.\hat{\mu} = \frac{1}{n} \sum_{i=1}^{n} s_i.

For independent, identically distributed prompt scores with finite variance, the standard error is estimated by σ^/n\hat{\sigma}/\sqrt{n}, where σ^\hat{\sigma} is the sample standard deviation of the scores.

A judge is useful only if it agrees with people, so its agreement with human ratings is measured on a sample, and its known biases are corrected. In pairwise comparisons, the judge is asked twice with the responses swapped, because judges favour particular positions; and scores are adjusted for length, because judges tend to prefer longer answers (Zheng et al., 2023; Dubois et al., 2024).

Key Papers

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Zheng et al.2023NeurIPS Datasets and Benchmarks
Read paper →

Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators

Dubois et al.2024COLM
Read paper →

Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference

Chiang et al.2024ICML
Read paper →

Connections

Prerequisites

Maximum LikelihoodCalibration