Jargon:Model collapse
1 min read
Say this at the party
“Model collapse is what can happen when AI learns mostly from other AI's output: like a photocopy of a photocopy, it slowly gets blander and more wrong.”
In plain English
AI models learn from huge piles of text and images, mostly scraped from the internet. More and more of the internet is now made by AI.
Researchers found that if a model trains on another model's output, again and again, it starts to forget the rare and unusual stuff. After many rounds it can produce nonsense.
Other researchers showed you can avoid this by keeping real human data in the mix rather than replacing it. So it's a real risk, not a certain doom.
Where it came from
The term was coined by Ilia Shumailov and colleagues in a 2023 paper called "The Curse of Recursion". A version was published in Nature in 2024.
See it in action
Imagine a model trained on dog photos where only a few are of rare breeds. Train a new model on its pictures, then another on those, and after a few rounds the rare breeds vanish and every dog looks like a golden retriever.
What people get wrong
- It doesn't mean today's chatbots are collapsing. AI companies filter their data and mix in human-made material.
- Using some AI-made data isn't automatically bad. Carefully chosen synthetic data is used all the time. The problem is replacing real data with it, round after round.
Go deeper
- PaperShumailov et al., Nature, 2024AI models collapse when trained on recursively generated dataThe peer-reviewed paper showing models degrade when trained on their own output.
- PaperShumailov et al., 2023The Curse of Recursion: Training on Generated Data Makes Models ForgetThe original preprint that coined the term model collapse.
- PaperGerstgrasser et al., 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic DataThe counterpoint: keep real data alongside AI data and collapse can be avoided.
- ArticleWikipediaModel collapseA short overview of the idea and the debate about how worried to be.
Related jargon
Drafted with AI. The AI Council (Claude, ChatGPT and Gemini) gave counsel. Phin signed it off.