Why doesn't approving every step keep me safe?
When every step needs approval, people stop reading.
Claude Code asks: "Allow this command?"
The first few times, you read every word. By the tenth, you skim. By the fiftieth, your finger is on Enter before the box has finished appearing.
Permission prompt: the pop-up that asks you to approve an action before the agent does it.
Anthropic looked at how people use these prompts in Claude Code. People approved 93% of them.
That isn't surprising. Most of what an agent asks to do is boring and fine: read a file, list a folder, run the tests.
The trouble is what happens to your attention when 93 out of 100 answers are "yes".
Anthropic ran a test with 1,053 paid testers. Partway through each session, one ordinary permission prompt was swapped for a clearly dangerous command.
Only 13.6% of people refused it.
The other 86% approved a dangerous action they'd been asked about directly.
Approval fatigue: when you're asked to confirm things so often that you stop really checking.
It's like a car alarm. If they go off all day for no reason, nobody looks up when a real one goes off.
And it's often worse than fatigue. Many people can't read the command anyway. "Allow find . -name '*.tmp' -delete?" means nothing if you don't know what it does.
There's an older name for the slow slide into trusting something because nothing bad has happened yet: the normalisation of deviance. The sociologist Diane Vaughan used it to explain the Challenger shuttle disaster in 1986.
Simon Willison applies it to AI: each time an unreliable tool gets away with it, we treat it as a bit safer, and approve a bit more without looking.
Fewer, better prompts. If safe actions are pre-approved, the prompts you do see deserve attention.
Auto mode. Anthropic's approach in Claude Code: a separate AI checker lets routine actions through and stops risky ones. In the same test, it would have blocked 89% of the dangerous commands. That still leaves 11%.
A sandbox. Sandbox: a fenced-off area where the agent can do what it likes but can't reach anything outside. Think of a playpen: it doesn't matter what the toddler does inside it.
Pre-approve the boring stuff (reading files, running tests) so the prompts that are left mean something.
Slow down on the scary words: delete, rm, drop, force, reset, overwrite (see "Before you let an AI agent loose").
Ask "what does this do?" whenever you can't read a command. It's a fair question every time.
Don't turn off permissions on your real computer. If you want it to run freely, give it a sandbox or a copy.