Jargon:Evals
Say it: EE-vals · 1 min read
Say this at the party
“Your own tests for whether an AI does your job well. Benchmarks are the public exam; evals are the driving test for your road.”
In plain English
"Eval" is short for evaluation. It's a test you write: give the AI an input, then check its answer against what good looks like.
Unlike a public benchmark, evals are about your task. A recipe app checks its AI never forgets allergies. A law firm checks it quotes the right clause.
You run them again every time you change the prompt or switch models, so you spot things getting worse before your customers do.
Where it came from
Testing models is as old as machine learning. The short word "evals" is now everyday talk among AI builders, helped by OpenAI's open-source Evals framework and by people like Hamel Husain, who argued in 2024 that failed AI products almost always lacked them.
See it in action
A small online shop uses AI to answer delivery questions. The owner writes 50 real customer questions with the right answers, and checks the bot passes them all before switching to a cheaper model.
What people get wrong
- Evals aren't the same as benchmarks. A model that tops the public charts can still fail your evals.
- They don't need to be fancy. A spreadsheet of real questions and good answers, checked regularly, beats "it looked fine when I tried it".
Go deeper
- PostHamel Husain, 2024Your AI Product Needs EvalsThe practical case for evals, with a real example and three levels of testing.
- PostAnthropic, 2026Demystifying evals for AI agentsHow Anthropic thinks about writing evals, starting from the basics.
- GuideAnthropic, Claude Platform DocsDefine success criteria and build evaluationsA how-to for writing your first evals for a Claude app.
- GuideOpenAIGitHub - openai/evals: Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.OpenAI's open-source tool for writing and running evals.
Related jargon
Drafted with AI. The AI Council (Claude, ChatGPT and Gemini) gave counsel. Phin signed it off.