Continuous Function

Foundation reference

Weak-to-Strong Generalization

Weak-to-strong generalization tests whether a stronger pretrained model can learn from labels produced by a weaker model and outperform that supervisor. It provides an experimental analogy for how people might supervise models more capable than themselves (Burns et al., 2024).

Concept 88 of 100Scaling & AlignmentStudy phase 12
Go to the mathAll concepts
Key equation
PGR=weak-to-strong−weakstrong ceiling−weak\text{PGR} = \frac{\text{weak-to-strong} - \text{weak}}{\text{strong ceiling} - \text{weak}}
Reading map and next steps

Study guide

Where to start

Concept 88 of 100Weak-to-StrongScaling & Alignment · Study phase 12: Advanced alignment & safety research
Question

How can a model trained to imitate wrong labels end up more accurate than the labels, and what does that say about what pretraining already gave it?

Go to the mathAll conceptsNext concept
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.

Pick a concrete input and say what the equation should give before you read the math.

CheckCheck against the math and the papers.

Work through the math, then read the cited papers for the original argument.

GeneralizeName what stays the same.

Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.

ContinueFollow the connections.

Use the prerequisites and related concepts at the end of the page to fill the next gap.

Why It Matters for Modern Models

  • Weak-to-strong generalization tests whether a stronger pretrained model can learn from labels produced by a weaker model and outperform that supervisor. It provides an experimental analogy for how people might supervise models more capable than themselves (Burns et al., 2024).
  • On NLP tasks, strong models trained on weak labels consistently beat their supervisors: naive fine-tuning typically recovered about half of the gap, and an auxiliary confidence loss nearly 80%, bringing a GPT-4-family model supervised by a GPT-2-level model close to GPT-3.5-level performance (Burns et al., 2024).
  • The effect depended on the task. For reward modeling, naive fine-tuning usually recovered only about 10% of the gap, which led the authors to warn that techniques like RLHF may scale poorly to models much stronger than their supervisors (Burns et al., 2024).

Questions to Think About

  • Why does a loss that encourages the strong model to stay confident, even when it disagrees with the weak labels, recover more of the gap?
  • In what ways is a small model supervising a large one a poor stand-in for people supervising a more capable model?

Core Math

Key Equation
PGR=weak-to-strong−weakstrong ceiling−weak\text{PGR} = \frac{\text{weak-to-strong} - \text{weak}}{\text{strong ceiling} - \text{weak}}

Burns et al. (2024) study an analogy for supervising models more capable than their supervisors: a weak model supervises a strong one.

  1. Fine-tune a small model on ground-truth labels; it becomes the weak supervisor WW.
  2. Use WW to label held-out examples, y~=W(x)\tilde{y} = W(x); many of these labels are wrong.
  3. Fine-tune a large pretrained model SS on the weak labels, minimizing E[ℓ(S(x),y~)]\mathbb{E}\big[\ell(S(x), \tilde{y})\big], and measure its accuracy against the true labels yy.

The result is compared with the weak supervisor and with a strong ceiling, the same large model fine-tuned on ground truth. The performance gap recovered is

PGR=weak-to-strong−weakstrong ceiling−weak\text{PGR} = \frac{\text{weak-to-strong} - \text{weak}}{\text{strong ceiling} - \text{weak}}

PGR = 0 means the strong model only matched its weak supervisor; PGR = 1 means it did as well as if it had been trained on ground truth.

Key Papers

Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

Burns et al.2024ICML
Read paper →

Connections

Prerequisites

RLHFKnowledge Distillation