Jargon:Red teaming
1 min read
Say this at the party
“Red teaming means deliberately attacking your own AI, trying to make it misbehave, so you can fix the problems before the public finds them.”
In plain English
A red team is a group whose job is to play the baddie. For AI, they try every trick to make it say something harmful, leak secrets or go off the rails.
Whatever they find gets fixed, and the AI is tested again. Some teams now use other AIs to come up with thousands of attacks automatically.
Think of it as hiring a burglar to test your locks.
Where it came from
From Cold War war games in the early 1960s, when the RAND Corporation ran simulations for the US military with the "red team" playing the Soviet Union. It spread to computer security, and in 2022 AI researchers, including a team at Anthropic, published papers on red teaming language models.
See it in action
Before a new chatbot launches, a team spends weeks trying to get it to give dangerous advice, write hateful jokes or reveal its hidden instructions, then hands the worst failures to the engineers.
What people get wrong
- Red teaming can't prove an AI is safe. It only finds the problems the testers thought to look for.
- It isn't only for hackers. Doctors, lawyers and ordinary users are often brought in to spot harms that experts miss.
Go deeper
- ArticleWikipediaRed teamWhere red teaming came from, from Cold War war games to cyber security.
- PaperGanguli et al., Anthropic, 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons LearnedAn early, detailed account of people red teaming chatbots and what they found.
- PaperPerez et al., 2022Red Teaming Language Models with Language ModelsUsing one AI to find the weak spots in another, automatically.
Related jargon
Drafted with AI. The AI Council (Claude, ChatGPT and Gemini) gave counsel. Phin signed it off.