Jargon:Distillation
1 min read
Say this at the party
“A big, clever AI teaches a small one by showing it lots of answers. The small one ends up nearly as good, but far cheaper and faster.”
In plain English
Big AI models are smart but slow and costly to run. Small ones are quick and cheap, but not as clever.
Distillation is a shortcut. You let the big model (the "teacher") answer lots of questions, and train the small model (the "student") to copy its answers.
The teacher's answers carry extra hints, like how sure it was about each option. So the student learns more than it would from a plain answer sheet.
Where it came from
Researchers were squeezing big models into small ones by 2006, when it was called "model compression". The word "distilling" comes from a 2015 paper by Geoffrey Hinton, Oriol Vinyals and Jeff Dean at Google.
See it in action
A company wants a speedy assistant on your phone. It asks its biggest model thousands of questions, then trains a tiny model on those replies so it can run without the cloud.
What people get wrong
- The student rarely matches the teacher on everything. It's usually a little worse, traded for being much faster and cheaper.
- It isn't the same as copying the teacher's insides. The student only learns from the teacher's answers, not by peeking at how it works.
Go deeper
- PaperHinton, Vinyals and Dean, Google, 2015Distilling the Knowledge in a Neural NetworkThe paper that named it, showing a big model's knowledge squeezed into a small one.
- PaperSanh et al., Hugging Face, 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighterA famous example: 40% smaller, 60% faster, keeping 97% of the skill.
- ArticleWikipediaKnowledge distillationClear overview, including the earlier history of model compression.
Related jargon
Drafted with AI. The AI Council (Claude, ChatGPT and Gemini) gave counsel. Phin signed it off.