Jargon:Transformer
1 min read
Say this at the party
“The design behind ChatGPT, Claude and Gemini. It reads a whole passage at once and works out which words matter most to each other.”
In plain English
Older language AI read text one word at a time, like someone reading through a keyhole. By the end of a long sentence it had often lost track of the start.
A transformer looks at all the words together. For each word it asks "which other words here help me understand this one?" That trick is called attention.
It also turned out to be very good at using lots of computer chips at once, which is how models got so big so fast. It's the T in GPT.
Where it came from
Eight researchers at Google introduced it in June 2017 in a paper called "Attention Is All You Need". It was built to improve machine translation, then took over almost all of AI.
See it in action
In "the trophy didn't fit in the suitcase because it was too big", a transformer links "it" to "trophy" by weighing every word in the sentence against every other.
What people get wrong
- Nothing to do with the robots in disguise, or the box on an electricity pole.
- A transformer is a design, not a product. ChatGPT, Claude and Gemini are all different models built on the same basic design.
Go deeper
- PaperVaswani et al., Google, 2017Attention Is All You NeedThe original paper. Possibly the most influential AI paper of the last decade.
- GuideJay AlammarThe Illustrated TransformerA famous picture-by-picture walk through how a transformer works.
- ArticleWikipediaTransformer (deep learning)History and detail, including the ideas that came just before it.
Related jargon
Drafted with AI. The AI Council (Claude, ChatGPT and Gemini) gave counsel. Phin signed it off.