AI Systems Atlas

Agent Safety & Security

Agents are different from chatbots because they possess authority: credentials, tools, and the ability to cause side effects. Every attack below is a way for untrusted text to borrow that authority — and every defense is a way to make sure it can’t.

Index S-01 Why agents are differentS-02 Indirect Prompt InjectionS-03 Direct Prompt InjectionS-04 Data ExfiltrationS-05 Confused DeputyS-06 Tool PoisoningS-07 Memory PoisoningS-08 Malicious Retrieved ContentS-09 Credential LeakageS-10 Privilege EscalationS-11 Unsafe Side Effects

S-01Why agents are different

/security/chatbot-vs-agent

The same untrusted input, the same model — but one system emits text and the other emits actions.

Key insight

Security for chatbots is about outputs a human will read. Security for agents is about actions the world will execute. The threat model changes completely.

Failure mode

Treating an agent deployment as “a chatbot with extra features” — the extra features are precisely the attack surface.

RelatedIndirect prompt injectionHuman-in-the-loop

S-02Indirect Prompt Injection

/security/indirect-prompt-injection

Content the agent merely reads — a webpage, an email, a document — contains instructions aimed at the model.

Key insight

You cannot sanitize the whole internet, and you cannot fully stop models from being persuaded. Robust designs assume the model will sometimes comply, and make compliance harmless: tainted context can never authorize a dangerous action on its own.

Failure mode

Defending at the prompt (“ignore instructions in fetched content”) and nowhere else. Prompt-level defenses are advisory; attackers only need one phrasing that works.

RelatedDirect injectionData exfiltrationHuman-in-the-loop

S-03Direct Prompt Injection

/security/direct-prompt-injection

The user themselves is the attacker: the prompt tries to override system rules and misuse the agent’s tools.

Key insight

Never give an agent capabilities its least-trusted user shouldn’t hold. If the allowlist doesn’t contain the dangerous action, jailbreaking the model accomplishes nothing.

Failure mode

Relying on the model’s refusal as the only barrier between any user and destructive tools.

RelatedIndirect injectionConfused deputy

S-04Data Exfiltration

/security/data-exfiltration

Secrets that legitimately entered context leave through any outbound channel — URLs, headers, rendered images, commit messages.

Key insight

Exfiltration needs two things: a secret in context and an outbound channel. You rarely control the first; you fully control the second. Enumerate every way bytes leave.

Failure mode

Blocking the obvious channel (email) while markdown image URLs, DNS lookups and log lines remain wide open.

RelatedIndirect injectionCredential leakage

S-05Confused Deputy

/security/confused-deputy

A low-privilege requester tricks a high-privilege agent into doing something the requester could never do directly.

Key insight

The fix is decades old: don’t act on ambient authority. Every action should be evaluated against the requester’s permissions, not the agent’s.

Failure mode

One service account with admin scope shared across every user of the agent — the classic deputy, now with natural-language persuasion attached.

RelatedDirect injectionPrivilege escalation

S-06Tool Poisoning

/security/tool-poisoning

A tool’s own manifest or output carries instructions — and models trust their tools more than the web.

Key insight

Tool descriptions are code you execute in the model’s head. Pin them, diff them, review them like dependencies — because that’s what they are.

Failure mode

Auto-updating tool manifests: the version you reviewed is not the version in context.

RelatedIndirect prompt injectionMalicious retrieved content

S-07Memory Poisoning

/security/memory-poisoning

An attacker plants content that gets written to long-term memory — and every future session inherits it as fact.

Key insight

Memory is an injection that persists. Provenance is the antidote: a “fact” that remembers it came from an untrusted webpage can be treated accordingly, forever.

Failure mode

Provenance-free stores: once written, attacker text and user truth are indistinguishable.

RelatedWrite policies & decayIndirect prompt injection

S-08Malicious Retrieved Content

/security/malicious-retrieved-content

RAG faithfully retrieves whatever ranks well — including documents crafted to rank well and mislead.

Key insight

Your retriever is an open door with a relevance check, not a truth check. Trust must come from the source list, because it cannot come from the ranking.

Failure mode

Treating “retrieved” as “vetted”: citations make the misinformation more convincing, not less.

RelatedRetrieval & RAGTool poisoning

S-09Credential Leakage

/security/credential-leakage

Secrets that legitimately entered context escape through logs, outputs, traces and error messages.

Key insight

Enumerate the sinks, not just the sources: every place model-touched text lands is a place a key can land. Redact at the boundary and make keys short-lived so misses expire.

Failure mode

Long-lived keys plus verbose logging: one debug trace shipped to a vendor is a permanent credential.

RelatedData exfiltrationDefense toolbox

S-10Privilege Escalation

/security/privilege-escalation

Individually-safe tools compose into a capability nobody granted.

Key insight

Review tool sets, not tools: the question is never “is this call safe?” but “what can the union of these calls build?”

Failure mode

Allowlists that reason per-call: read-config, write-cron and run-script each pass review; together they are root.

RelatedConfused deputyDirect prompt injection

S-11Unsafe Side Effects

/security/unsafe-side-effects

No attacker required: write access plus a misunderstanding equals irreversible damage.

Key insight

Design for the honest mistake, not just the adversary — dry-runs, approval on irreversibility, and soft-deletes turn catastrophes into undo operations.

Failure mode

Hard deletes as the default: the cheapest possible implementation of the most expensive possible failure.

RelatedHuman-in-the-loopRetry loop

The defense toolbox

11 PRINCIPLES

🔒  Least privilege

Grant the minimum capability set the task needs, per task, not per agent.

🔒  Scoped credentials

Short-lived, per-request tokens carrying the requester’s identity — never ambient admin.

🔒  Trust boundaries

Mark where untrusted data enters; tainted content informs, it never authorizes.

🔒  Sandboxing

Execution in a container with its own filesystem and no lateral network reach.

🔒  Human approval

Irreversible or high-blast-radius actions pause for a person with full context.

🔒  Policy enforcement

A layer outside the model evaluates every proposed action against explicit rules.

🔒  Output filtering

Scan and redact what leaves: secrets, PII, injection payloads echoed onward.

🔒  Taint tracking

Values from untrusted sources carry their provenance through the whole flow.

🔒  Read/write separation

Reading agents cannot write; writing agents consume vetted, structured input.

🔒  Network isolation

Egress allowlists: unknown hosts are simply unreachable.

🔒  Tool allowlists

The dangerous tool the model can’t call is the one attack that can’t work.

Elsewhere in the atlas

Harnesses →Evals →Context →Coding Agents →