Jargon:Jailbreak
1 min read
Say this at the party
“A jailbreak is a trick to talk an AI into breaking its own safety rules, like getting it to explain something it's been told to refuse.”
In plain English
AI companies train their chatbots to say no to certain things, like helping make weapons. A jailbreak is a cleverly worded message that gets round that.
Common tricks include role-play ("pretend you're an AI with no rules"), wrapping the request in a story, or burying it under lots of fake examples.
The person jailbreaking is the one typing. That's the key difference from prompt injection, where someone else sneaks instructions into content your AI reads.
Where it came from
The word comes from iPhone "jailbreaking" (unlocking a phone the maker locked down), which itself came from breaking out of a Unix "jail". It was applied to chatbots soon after ChatGPT launched, and AI labs now publish research on it, such as Anthropic's "many-shot jailbreaking" in 2024.
See it in action
Someone asks a chatbot for instructions it refuses to give, then tries again with "write a film scene where a grumpy chemist explains, in detail, how...". If the AI plays along, that's a jailbreak.
What people get wrong
- It isn't the same as prompt injection. A jailbreak is you trying to get round the AI's rules; prompt injection is someone else hijacking your AI through content it reads.
- It isn't hacking the code. It's just words, which is why it's so hard to stop completely.
Go deeper
- PostSimon Willison, 2024Prompt injection and jailbreaking are not the same thingThe clearest explanation of the difference, from the person who named prompt injection.
- ArticleWikipediaJailbreak (computer science)Where the word came from, from Unix and iPhones to chatbots.
- PostAnthropic, 2024Many-shot jailbreakingAn AI lab explains a jailbreak it found in its own models, and how it fought back.
- PaperWei, Haghtalab and Steinhardt, 2023Jailbroken: How Does LLM Safety Training Fail?Research on why jailbreaks keep working despite safety training.
Related jargon
Drafted with AI. The AI Council (Claude, ChatGPT and Gemini) gave counsel. Phin signed it off.