Jargon:Benchmark
1 min read
Say this at the party
“A public exam that every AI sits, so companies can compare scores. Handy for league tables, but a high score doesn't mean it suits your job.”
In plain English
A benchmark is a fixed, shared set of questions or tasks with known right answers. Anyone can run their AI on it and report the score.
That makes it easy to compare models side by side, like exam results. When a company says its new model is "state of the art", it usually means it beat others on some benchmarks.
Benchmarks get stale. Once models score nearly full marks, researchers say it's "saturated" and build a harder one.
Where it came from
Shared tests have long been used to compare computer programs. For today's chatbots, a famous one is MMLU, published in 2020 by Dan Hendrycks and colleagues: 57 subjects, from maths and law to US history.
See it in action
A new model launches with a chart showing it scored top marks on a maths benchmark. A teacher then tries it on her own Year 9 worksheets to see if it's actually better for her.
What people get wrong
- Models can accidentally see the test questions during training ("contamination"), so a high score can partly be memory, not skill.
- A benchmark tells you how a model does on that test, not on your work. To check your own task, you need evals.
Where Ugenie fits
Ugenie's AI Council asks Claude, ChatGPT and Gemini the same question and flags where they disagree, which can tell you more about your actual question than any leaderboard.
Get Ugenie →Go deeper
- PaperHendrycks et al., 2020Measuring Massive Multitask Language UnderstandingThe MMLU paper: one of the most quoted AI benchmarks.
- PaperChiang et al., 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human PreferenceA different kind of benchmark: people vote on which of two answers is better.
- ArticleWikipediaLanguage model benchmarkHow benchmarks are built, and problems like contamination and saturation.
Related jargon
Drafted with AI. The AI Council (Claude, ChatGPT and Gemini) gave counsel. Phin signed it off.