AI Systems Atlas

S-11Unsafe Side Effects

/security/unsafe-side-effects

No attacker required: write access plus a misunderstanding equals irreversible damage.

Step-by-step walkthrough2 variants · 6 steps

Failure

  1. The agent believes it’s doing what was asked.
  2. Write access plus a misunderstanding is the whole failure.
  3. By the time anyone notices, there is nothing to roll back.

Defended

  1. Destructive intent triggers a dry-run: the exact consequences, before any of them.
  2. A human sees “40,000 rows” — the number the agent’s misunderstanding hid.
  3. Even approved deletes are soft: reversible for 30 days. Irreversibility is a choice, not a default.
Key insight

Design for the honest mistake, not just the adversary — dry-runs, approval on irreversibility, and soft-deletes turn catastrophes into undo operations.

Failure mode

Hard deletes as the default: the cheapest possible implementation of the most expensive possible failure.

Further reading
RelatedHuman-in-the-loopRetry loop

Elsewhere in the atlas

Harnesses →Evals →Context →Coding Agents →Tools →Ops →