Jargon:Scaling laws
1 min read
Say this at the party
“The pattern that AI gets predictably better when you make it bigger and feed it more data and computing power. It's why firms spend billions.”
In plain English
Researchers noticed something neat. If you grow an AI model, give it more to read and more computer time, its mistakes shrink in a smooth, predictable way.
That means you can test small models, draw the line, and guess how good a giant one will be before you build it.
It's called a "law", but it's really an observed trend, like noticing that taller people tend to have bigger shoes.
Where it came from
A 2020 paper by Jared Kaplan and colleagues at OpenAI, "Scaling Laws for Neural Language Models", made the idea famous. In 2022, DeepMind's Chinchilla paper refined it, showing many models had been made too big for the amount of data they were given.
See it in action
Before spending a fortune on a new model, a lab trains a handful of small ones, plots how they improve, and uses the trend to choose how big to go and how much text it needs.
What people get wrong
- They don't promise the trend lasts forever. Data, money and electricity all run out somewhere, and nobody knows exactly where the curve bends.
- Lower error on a test isn't the same as being smarter at everything. Some skills improve smoothly, others don't.
Go deeper
- PaperKaplan et al., OpenAI, 2020Scaling Laws for Neural Language ModelsThe paper that put the idea on the map.
- PaperHoffmann et al., DeepMind, 2022Training Compute-Optimal Large Language ModelsThe "Chinchilla" paper: more data matters as much as more size.
- PostRich Sutton, 2019The Bitter LessonA short, famous essay on why methods that use more computing power keep winning.
- ArticleWikipediaNeural scaling lawA broad overview of what gets scaled and what has been found.
Related jargon
Drafted with AI. The AI Council (Claude, ChatGPT and Gemini) gave counsel. Phin signed it off.