A self-improvement loop uses a model's outputs to train its next version, with a check deciding which outputs enter the training set. Filtering a model's own outputs for correctness turns a few worked examples into a large training set: STaR beat a model fine-tuned to predict answers directly on several datasets, and on CommonsenseQA performed comparably to fine-tuning a 30-times larger model (Zelikman et al., 2022).
Concept 98 of 100Scaling & AlignmentStudy phase 13
Concept 98 of 100Self-ImprovementScaling & Alignment · Study phase 13: Recent research (2024–2025)
Question
If the model only trains on answers it already produced, how can it get better at problems it could not solve before, and what role does retrying with the correct answer as a hint play (Zelikman et al., 2022)?
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
A self-improvement loop uses a model's outputs to train its next version, with a check deciding which outputs enter the training set. Filtering a model's own outputs for correctness turns a few worked examples into a large training set: STaR beat a model fine-tuned to predict answers directly on several datasets, and on CommonsenseQA performed comparably to fine-tuning a 30-times larger model (Zelikman et al., 2022).
Reinforcement learning on problems with checkable answers can teach reasoning without human-written reasoning traces: DeepSeek-R1-Zero was trained by large-scale reinforcement learning with no supervised fine-tuning first, and DeepSeek-R1's reasoning was then distilled into six smaller models of 1.5 to 70 billion parameters based on Qwen and Llama (Guo et al., 2025).
A loop can only reinforce what its check rewards: a verifier that accepts flawed reasoning with a correct final answer teaches flawed reasoning, and with no filter at all, training on self-generated data loses diversity over generations (see Model Collapse).
Questions to Think About
Why does SPIN stop improving once the model's answers can no longer be told apart from the human ones, and what does that imply about its ceiling (Chen et al., 2024)?
Why might a small model fine-tuned on a large model's reasoning traces outperform the same small model trained directly with reinforcement learning, as Guo et al. (2025) report?
Core Math
Key Equation
θk+1=argθminE(x,y)∼Dk+1[−logπθ(y∣x)]
A self-improvement loop uses a model's own outputs to train its next version. A common form generates answers, keeps those that pass a check, and fine-tunes on them:
Dk+1=Dk∪{(x,y^):y^∼πθk(⋅∣x),V(x,y^)=1}
θk+1=argθminE(x,y)∼Dk+1[−logπθ(y∣x)]
Here V is a verifier, for example a check of the final answer against a known solution (Zelikman et al., 2022). Without the filter, the loop has nothing to learn from except its own distribution (see Model Collapse).
Self-play fine-tuning (SPIN) uses human-written answers instead of a verifier. Each round trains the model to prefer the human answer y over its own previous answer y′, with a logistic loss on log-probability ratios like that of DPO; the objective is optimal only when the model's distribution matches the human data (Chen et al., 2024).
Distillation passes what a large teacher model has learned to a smaller student, often by fine-tuning the student on the teacher's sampled outputs: