Thursday, November 12, 2026

What Is Model Collapse? Why AI Trained on AI Data Degrades

Say a company trains a writing model, publishes millions of its outputs on the web, and two years later scrapes the web to train the next version. Some of what the new model learns from is its own earlier writing. Repeat that a few times, and each generation is learning less from people and more from copies of copies.

That loop has a name. Model collapse is the gradual loss of quality and diversity that happens when generative models are trained on data produced by earlier models instead of on real data. Rare cases disappear first, then outputs drift and narrow until they no longer reflect the original data. The best-known evidence is a 2024 Nature paper by Ilia Shumailov and colleagues, and later work shows how to avoid it: keep real data in the training mix instead of replacing it. This article explains what model collapse is, why it happens, how to see it in a 24-line simulation, and what it means for anyone training or fine-tuning models.

Key Takeaways

  • Model collapse is a degenerative process in which models trained on recursively generated data lose the rare parts of the original data first and then drift away from it.
  • Shumailov and colleagues showed the effect in language models, variational autoencoders and Gaussian mixture models in “AI Models Collapse When Trained on Recursively Generated Data” (Nature, 2024).
  • The cause is compounding error: every generation samples a finite amount of data and fits it imperfectly, and those errors add up instead of canceling out.
  • Collapse is not inevitable. Research by Gerstgrasser and colleagues found that accumulating synthetic data alongside the original real data, instead of replacing it, prevents it in their experiments.
  • For practitioners, the lessons are to keep real, human-made data in the mix, track where training data comes from, and measure diversity, not only accuracy.

What Is Model Collapse?

Model collapse is what happens when each new generation of a model is trained on data produced by the previous generation. Over time the model forgets the true data distribution. It first loses the tails, meaning rare words, unusual cases and minority patterns, and eventually produces narrow, repetitive output that has little to do with the original data.

The term comes from AI Models Collapse When Trained on Recursively Generated Data, published in Nature in July 2024 by Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson and Yarin Gal. An earlier version appeared in 2023 as the preprint The Curse of Recursion: Training on Generated Data Makes Models Forget.

The paper describes two stages:

  • Early model collapse. The model starts losing information about the tails of the distribution. Overall metrics can look fine or even improve, while performance on rare cases quietly gets worse.
  • Late model collapse. The model’s output converges to something that looks very little like the original data, often with much less variety.

Model collapse is different from mode collapse in generative adversarial networks (GANs). Mode collapse is a training failure inside a single GAN, where the generator produces only a few kinds of output. Model collapse happens across generations of models, through the data they pass on. Our comparison of diffusion models, GANs and VAEs covers how each of those model types works.

Why Does Model Collapse Happen?

Shumailov and colleagues identify three sources of error that compound from one generation to the next:

  • Statistical approximation error. Each generation trains on a finite sample. Rare events may not appear in the sample at all, and once they are missing, the next model cannot learn them back. This is the main cause.
  • Functional expressivity error. A model can only represent certain kinds of distributions. If the true distribution is more complex, the model’s version is an approximation, and the next generation learns from that approximation.
  • Functional approximation error. Training methods are imperfect. Choices like the loss function and optimizer add their own bias to what the model learns.

Any one of these alone causes only a minor error. The problem is that each generation inherits the previous generation’s errors and adds its own.

See Model Collapse in 24 Lines of Python

The simplest case in the paper is a single Gaussian (bell curve) distribution. The simulation below “trains” a model by estimating the mean and standard deviation of its data, then generates the next generation’s data by sampling from that model. It compares two policies: replacing the data with synthetic samples each generation, and accumulating synthetic samples on top of the original real data.

import numpy as np

rng = np.random.default_rng(0)
N = 100            # samples per generation
GENERATIONS = 500

real = rng.normal(0.0, 1.0, N)   # generation 0: "human" data from N(0, 1)


def run(accumulate: bool) -> list[float]:
    data = real.copy()
    stds = []
    for gen in range(1, GENERATIONS + 1):
        mu, sigma = data.mean(), data.std()          # "train" the model
        synthetic = rng.normal(mu, sigma, N)         # sample from it
        data = np.concatenate([data, synthetic]) if accumulate else synthetic
        stds.append(sigma)
    return stds


replace = run(accumulate=False)
keep = run(accumulate=True)
for gen in (1, 10, 100, 250, 500):
    print(f"generation {gen:3}: replace std = {replace[gen-1]:.3f}   accumulate std = {keep[gen-1]:.3f}")

Output:

generation   1: replace std = 0.962   accumulate std = 0.962
generation  10: replace std = 0.676   accumulate std = 0.973
generation 100: replace std = 0.267   accumulate std = 0.960
generation 250: replace std = 0.030   accumulate std = 0.959
generation 500: replace std = 0.010   accumulate std = 0.958

The true standard deviation is 1.0. When each generation trains only on the previous generation’s samples, the spread shrinks to almost nothing: by generation 500 the “model” produces values packed tightly around one point. The rare values far from the center, the tails, were the first to go. When the original real data stays in the pool, the estimate stays close to the truth. The exact numbers depend on the random seed, but the pattern does not.

What Model Collapse Looks Like in Language Models

The Nature paper also tested a real language model. The authors fine-tuned Meta’s OPT-125m model on a Wikipedia-based dataset, used it to generate a new dataset, fine-tuned the next model on that, and repeated. Over generations, perplexity on the original data rose, which means the models got worse at predicting real text, and the outputs grew more repetitive.

One example from the paper became widely quoted. Given a prompt about medieval church architecture, a later-generation model drifted into an unrelated list of jackrabbits with different-colored tails. The authors also found that keeping a share of the original human data in each generation’s training set reduced the damage considerably.

Is Model Collapse Inevitable?

No, and this is the most useful finding for practitioners. In Is Model Collapse Inevitable? (2024), Matthias Gerstgrasser and colleagues compared two settings. When each generation’s data replaced the previous data, models collapsed, matching the earlier results. When synthetic data was added to the existing real data instead, so the pool grew over time, test error stayed bounded and collapse did not occur. They found this across language models, diffusion models for molecules and VAEs for images.

The simulation above shows the same contrast. Collapse comes from losing touch with real data, not from the mere presence of synthetic data.

That matters because synthetic data is used on purpose all the time. Model developers use larger models to generate training data for smaller ones, a process called distillation. Meta’s Llama 3.3 model card, for example, says its license allows using the model’s outputs to improve other models, including synthetic data generation and distillation. Done carefully, with filtering, checks for quality and plenty of real data, this is a standard technique, not a path to collapse.

What Model Collapse Means for the Web

The broader worry is about the public web as a training source. As more text and images online are generated by AI, future web scrapes will contain more model output, often unlabeled. The Nature authors argue that data about genuine human interactions will become more valuable as a result, and that the AI community may need ways to track where content comes from.

This is a real risk to manage, not a sudden cliff. Training data is usually filtered and deduplicated, older human-written archives remain available, and as the accumulation research shows, adding synthetic data to real data is far less harmful than replacing it.

How to Avoid Model Collapse in Your Own Work

If you fine-tune models, generate synthetic training data, or build systems that learn from their own outputs, these habits help. They apply whether you work with a hosted API or one of the open-weight families covered in our guide to comparing Gemma, Llama and Mistral.

  1. Keep real data in every training round. Add synthetic data to a fixed base of human-made data. Do not replace it.
  2. Track provenance. Record which examples are human-made, which are synthetic and which model produced them. You cannot control the mix if you do not know it.
  3. Filter synthetic data. Check it against rules, tests or human review before training on it. Drop duplicates and near-duplicates.
  4. Measure diversity and rare cases. Track vocabulary variety, the spread of outputs and accuracy on uncommon inputs, not only average scores. Early collapse hides behind good averages.
  5. Watch feedback loops. A recommendation or content system that retrains on what it showed users can narrow over time in the same way. Keep some fresh, independent data coming in.

Back to the Copy of a Copy

The company from the opening does not have to stop using its own outputs. It has to stop treating them as a replacement for real data. If each new model trains on a pool where human-written text stays in place, synthetic text is filtered and labeled, and rare cases are measured, the loop that causes collapse is broken. The copies can help, as long as the original stays in the room.

Frequently Asked Questions

What Is Model Collapse in Simple Terms?

It is what happens when AI models learn mostly from other AI models’ output instead of real data. Each generation loses a little of the variety in the original data, starting with rare cases. After enough generations, the output becomes narrow, repetitive and inaccurate.

Who Discovered Model Collapse?

The term and the main evidence come from Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson and Yarin Gal. Their work appeared as a 2023 preprint, “The Curse of Recursion”, and in Nature in 2024 as “AI Models Collapse When Trained on Recursively Generated Data”.

Is Model Collapse the Same as Mode Collapse?

No. Mode collapse is a failure during the training of a single GAN, where the generator produces only a few kinds of output. Model collapse happens across several generations of models trained on each other’s data.

Is It Safe to Train on Synthetic Data?

It can be, with care. Research shows the danger comes from replacing real data with synthetic data over many rounds. Keeping real data in the mix, filtering synthetic examples and tracking where data came from make synthetic data a useful tool rather than a risk.

Disclaimer: The content on this website reflects the views of contributing authors and not necessarily those of Generative AI Lab. This site may contain sponsored content, affiliate links, and material created with generative AI. Thank you for your support.

Roberto Iriondo, founder of Generative AI Lab, at Google Cloud Next in Las Vegas
Roberto Iriondo
Roberto Iriondo is the founder of Generative AI Lab. He has spent ten years in marketing for AI companies and research teams, including technical content at Cohere and Snorkel AI, after starting out as a front end engineer in Carnegie Mellon University's Machine Learning Department. He co-founded Towards AI. More about Roberto

Must read

- Advertisement -spot_img

More articles

- Advertisement -spot_img

Latest articles

Available for Amazon Prime