Synthetic data can stand in for expensive human annotation: Self-Instruct used GPT-3 to generate its own instruction-following examples and improved it by 33 percentage points on Super-NaturalInstructions, on a par with InstructGPT-001 (Wang et al., 2023).
Concept 77 of 100Scaling & AlignmentStudy phase 11
Concept 77 of 100Synthetic DataScaling & Alignment · Study phase 11: Reasoning at inference time & synthetic data
Question
If a model generates its own training data, where does the new information come from: the teacher, the filter, or the correct answers used to check the outputs?
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Synthetic data can stand in for expensive human annotation: Self-Instruct used GPT-3 to generate its own instruction-following examples and improved it by 33 percentage points on Super-NaturalInstructions, on a par with InstructGPT-001 (Wang et al., 2023).
Selection can matter as much as scale: phi-1, a 1.3-billion-parameter code model trained on 6 billion tokens of filtered web data and 1 billion tokens of textbooks and exercises generated with GPT-3.5, reached 50.6% pass@1 on the HumanEval coding benchmark (Gunasekar et al., 2023).
Projections suggest that the largest training runs will use roughly the whole stock of public human-written text sometime between 2026 and 2032; synthetic data is one proposed response, alongside transfer from other data sources and more data-efficient training (Villalobos et al., 2024).
Questions to Think About
Why does training a smaller model on a stronger model's outputs work well, while training a model only on its own unfiltered samples tends to lose the rare cases (Shumailov et al., 2024)?
How should synthetic data be mixed with real data, and what does a benchmark score mean if the model that generated the data may have seen the benchmark?
Core Math
Key Equation
θt+1=Train(θt,Filter(Generate(θt)))
Synthetic data is training data produced by a model instead of collected from people. A common recipe has three steps:
Generate candidates: y∼pθ(y∣x) for prompts x, often from a stronger "teacher" model.
Filter for quality: keep Dsynth={(xi,yi):V(xi,yi)>τ}, where V is a verifier, a reward model or a consistency check.
Train on what is kept, usually mixed with real data: θ′=argminθL(θ;Dreal∪Dsynth).
When the generating model is the model being trained, the steps form a loop:
θt+1=Train(θt,Filter(Generate(θt)))
STaR is an example: the model writes rationales for questions, keeps only those that reach the correct answer, fine-tunes on them and repeats (Zelikman et al., 2022).
New information enters through the teacher or through the filter. A model trained only on its own unfiltered samples has nothing new to learn from, and repeating that loses the tails of the original distribution (Shumailov et al., 2024).