Jargon:Token
1 min read
Say this at the party
“A token is a bite-sized chunk of text, often part of a word. AIs read, write and charge you in tokens, not words.”
In plain English
An AI doesn't see words the way you do. Before it reads anything, your text is chopped into small pieces called tokens.
A token might be a whole short word like "cat", a chunk of a longer word like "un" and "believ" and "able", or a single letter or symbol. In English, one token is roughly three or four letters.
Tokens are how AI is measured. How much a model can hold in its head at once, how fast it replies and how much it costs are all counted in tokens.
Where it came from
"Token" is an old word in computing for a single unit of text. The way modern AIs split words into pieces comes from byte-pair encoding, a data-squashing trick Philip Gage described in 1994. In 2015, Rico Sennrich and colleagues showed it could break rare words into smaller parts for translation, and language models took it from there.
See it in action
Priya pastes a long report into Claude and gets a warning that it's too long. The report isn't too many pages for her. It's too many tokens for the model's context window.
What people get wrong
- A token is not the same as a word. Long or unusual words, names and other languages often take several tokens each.
- The same text can be a different number of tokens in different AIs, because each one chops text up in its own way.
Go deeper
- GuideAnthropicGlossary - Claude Platform DocsAnthropic's own short definition of tokens, with how many letters a token roughly covers.
- GuideHugging Face, LLM CourseTokenizers · Hugging FaceA friendly walk-through of how text gets split into word, letter and part-word tokens.
- PaperSennrich, Haddow and Birch, 2015Neural Machine Translation of Rare Words with Subword UnitsThe paper that showed splitting words into pieces works, the trick behind today's tokens.
- ArticleWikipediaByte-pair encoding - WikipediaThe 1994 compression trick that most AI tokenisers are built on.
Related jargon
Drafted with AI. The AI Council (Claude, ChatGPT and Gemini) gave counsel. Phin signed it off.