AI Systems Atlas

S-03Direct Prompt Injection

/security/direct-prompt-injection

The user themselves is the attacker: the prompt tries to override system rules and misuse the agent’s tools.

Step-by-step walkthrough2 variants · 5 steps

Attack

  1. The attacker is the user: the prompt itself tries to override the system rules.
  2. If the model complies, the injected goal becomes a real tool call.

Defended

  1. Same hostile prompt. The model can still be fooled…
  2. …but it can only propose. Enforcement lives outside the model.
  3. Destructive commands are simply not in the allowlist. Compliance without capability is harmless.
Key insight

Never give an agent capabilities its least-trusted user shouldn’t hold. If the allowlist doesn’t contain the dangerous action, jailbreaking the model accomplishes nothing.

Failure mode

Relying on the model’s refusal as the only barrier between any user and destructive tools.

Further reading
RelatedIndirect injectionConfused deputy

Elsewhere in the atlas

Harnesses →Evals →Context →Coding Agents →Tools →Ops →