Jargon:Inference
1 min read
Say this at the party
“Training is when an AI learns. Inference is when it actually uses what it learned to answer you. Every reply you get is inference.”
In plain English
There are two stages in an AI model's life. First it's trained, which is slow and expensive and happens once (or now and then).
After that, every time someone asks it something, the model runs and produces an answer. That running is called inference.
It's like the difference between going to cooking school and cooking dinner. You only train once, but you cook every night, and that's where the energy bills add up.
Where it came from
"Inference" is an old word for reaching a conclusion from evidence. In AI it became the standard name for running a trained model on new data, as opposed to training it.
See it in action
Sam asks Claude to turn a messy shopping list into a meal plan. The few seconds Claude spends writing the answer is inference, running on computers in a data centre.
What people get wrong
- The model doesn't learn from you during inference. Its settings are fixed. Any memory you see is a separate feature built around it.
- Inference isn't cheap just because it's quick. Millions of people asking questions every day can cost more than training did.
Go deeper
- GuideDave Bergmann, IBM, 2024What is AI Inference?A clear explainer of inference and how it differs from training.
- PostNVIDIAWhat’s the Difference Between Deep Learning Training and Inference?The chip maker's own plain take on training versus inference.
- GuideGoogle for DevelopersMachine Learning GlossaryGoogle's short definitions of inference for both classic ML and language models.
Related jargon
Drafted with AI. The AI Council (Claude, ChatGPT and Gemini) gave counsel. Phin signed it off.