Training on model-generated data can help or hurt; retraining only on a model's own samples gradually loses the tails of the distribution, which Shumailov et al. (2024) observed in language models, variational autoencoders and Gaussian mixture models.
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Training on model-generated data can help or hurt; retraining only on a model's own samples gradually loses the tails of the distribution, which Shumailov et al. (2024) observed in language models, variational autoencoders and Gaussian mixture models.
As more text on the web is written by language models, training sets crawled from the web will contain earlier models' output; Shumailov et al. (2024) argue that data from genuine human interactions will become more valuable as a result.
How much real data is kept matters. In Shumailov et al.'s language-model experiments, keeping 10% of the original data in each generation led to only minor degradation, and analyses of retraining with real data retained suggest that collapse is avoidable when real data is not thrown away (Bertrand et al., 2024; Gerstgrasser et al., 2024).
Questions to Think About
In the Gaussian example the variance estimate is unbiased at every step; how can the fitted variance still drift toward zero over many generations?
Why does accumulating synthetic data alongside the original real data avoid collapse, when replacing the real data does not (Gerstgrasser et al., 2024)?
Core Math
Key Equation
pt=(1−α)p∗+αpθt
Model collapse is the degradation that occurs when generative models are trained, generation after generation, on data produced by earlier models (Shumailov et al., 2024).
Let p∗ be the real data distribution and pθt the model at generation t. Suppose each generation is trained on a mixture in which a fraction α of the data is synthetic:
pt=(1−α)p∗+αpθt,θt+1=Fit(samples from pt)
α=1, replacing the real data. Each model is fitted only to samples from its predecessor. Errors from finite samples and from the model's own approximations accumulate, and the tails of p∗ disappear first (Shumailov et al., 2024).
α<1, keeping real data. Bertrand et al. (2024) prove that iterative retraining is stable if the first model is close enough to p∗ and the fraction of real data is large enough. Gerstgrasser et al. (2024) show that accumulating synthetic data alongside all of the original real data, instead of replacing it, keeps test error bounded in a linear model.
A Gaussian example. Fit a mean and variance to M samples, draw M new samples from the fitted Gaussian, refit, and repeat. With the unbiased variance estimate, each refit multiplies the variance by a random factor χM−12/(M−1). The factor has mean 1, but its logarithm has a negative mean (Jensen's inequality), so over many generations the fitted variance drifts towards zero while the mean wanders: the fitted distribution moves away from the original and loses its tails.
Key Papers
AI models collapse when trained on recursively generated data