Jargon:Mixture of experts
1 min read
Say this at the party
“A mixture of experts is an AI split into lots of smaller specialist parts, and only a few of them wake up for each word. Big brain, small bill.”
In plain English
Normally, every part of an AI works on every word it reads or writes. That gets slow and expensive as models grow.
A mixture of experts splits part of the model into many smaller sections, called experts. A little traffic controller, called a router, sends each word to just a couple of them.
So the model can be huge in total but only use a slice of itself at a time, which makes it faster and cheaper to run.
Where it came from
The idea dates back to a 1991 paper, "Adaptive Mixtures of Local Experts", by Robert Jacobs, Michael Jordan, Steven Nowlan and Geoffrey Hinton. In 2017, Noam Shazeer and colleagues at Google showed it could work at huge scale, and open models like Mixtral brought it into the spotlight in late 2023.
See it in action
A company runs a mixture of experts model on its servers. It has the knowledge of a very big model but answers customers almost as quickly and cheaply as a small one, because most of it sits idle for each word.
What people get wrong
- The experts aren't a doctor, a lawyer and a chef. Research found they tend to specialise in things like punctuation or names, not school subjects.
- It isn't several separate AIs chatting together. It is one model that only switches on part of itself at a time.
Go deeper
- PostSanseviero et al., Hugging Face, 2023Mixture of Experts ExplainedThe go-to explainer: history, how routing works, and what experts actually learn.
- PaperShazeer et al., Google, 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerThe paper that scaled the idea up for modern AI.
- ArticleWikipediaMixture of experts - WikipediaBackground and history, back to the 1991 original.
Related jargon
Drafted with AI. The AI Council (Claude, ChatGPT and Gemini) gave counsel. Phin signed it off.