Jargon:Computer use
2 min read
Say this at the party
“When an AI works a computer the way you do: looking at the screen, moving the mouse, clicking and typing.”
In plain English
Computer use means an AI works a computer the way you do. It looks at the screen, moves the pointer, clicks and types. There's no special back door. It presses the same buttons you would.
It works like a flipbook. The AI takes a screenshot (a picture of the screen) and works out what's there. It counts the pixels (the tiny dots on the screen) to the right spot and clicks. Then it takes another picture and goes again. Anything that flashes up and vanishes quickly can slip past it.
Why bother? Most AI helpers can only use tools built specially for them. This one can, in theory, use any app a person can. In 2024 the best AI managed about 12% of the tasks on the main test, against over 72% for people. By 2026 Anthropic said it was reaching human level on some jobs, like tricky spreadsheets and long web forms.
Where it came from
Researchers were teaching AI to do tasks on websites with keyboard and mouse actions back in 2017, in a paper called World of Bits (one of its authors was Andrej Karpathy). Anthropic gave the idea its name on 22 October 2024, launching "computer use" for Claude 3.5 Sonnet as an early test for developers. OpenAI followed with its own version, Operator, in January 2025.
See it in action
On the train, you ask Claude on your phone to put together a morning briefing. On your Mac at home, it checks your calendar and messages and pulls it all into one page. It uses a direct link to an app where there is one, and only clicks around the screen when there isn't.
What people get wrong
- It isn't a robot arm or a hacker inside your machine. It only sees screenshots and sends clicks and key presses, like an invisible person at the keyboard.
- It's not usually the best way for an AI to get things done. Clicking through the screen is slower than a direct link between apps, so it's meant as a fallback.
- "Human-level" means certain tasks, like filling in a long web form. It doesn't mean it can do your whole job.
- Watching it isn't enough to keep you safe. A web page can hide instructions that trick it, so start with apps you trust and keep sensitive data well away.
Go deeper
- PostAnthropic, 2024Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 HaikuThe launch that gave the idea its name, with an honest look at how far off human level it was.
- PostAnthropic, 2024Developing a computer use modelHow screenshots and pixel counting work, plus the demo where Claude wandered off to look at Yellowstone.
- PaperShi, Karpathy, Fan, Hernandez and Liang, 2017World of Bits: An Open-Domain Platform for Web-Based AgentsThe early research on AI clicking and typing its way round websites.
- PaperXie et al., XLANG Lab, 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer EnvironmentsThe main test everyone uses to score how well AI can work a real computer.
- PostAnthropic, 2026Put Claude to work on your computerThe everyday version: give Claude a job from your phone and it does it on your computer.
Related jargon
Drafted with AI. The AI Council (Claude, ChatGPT and Gemini) gave counsel. Phin signed it off.