AI Systems Atlas

S-06Tool Poisoning

/security/tool-poisoning

A tool’s own manifest or output carries instructions — and models trust their tools more than the web.

Step-by-step walkthrough2 variants · 7 steps

Attack

  1. A tool is installed from a registry — its manifest is just text.
  2. Hidden instructions ride in the description, loaded straight into context.
  3. Models weight tool descriptions heavily — they’re supposed to follow them.
  4. The poisoned description becomes a real call.

Defended

  1. Same registry — but manifests are pinned to a reviewed hash.
  2. A malicious update is a visible diff, not a silent context change.
  3. The model only ever loads a manifest a human has seen.
Key insight

Tool descriptions are code you execute in the model’s head. Pin them, diff them, review them like dependencies — because that’s what they are.

Failure mode

Auto-updating tool manifests: the version you reviewed is not the version in context.

Further reading
RelatedIndirect prompt injectionMalicious retrieved contentMCP architecture

Elsewhere in the atlas

Harnesses →Evals →Context →Coding Agents →Tools →Ops →