AI Systems Atlas

Agent Safety & Security

Agents are different from chatbots because they possess authority: credentials, tools, and the ability to cause side effects. Every attack below is a way for untrusted text to borrow that authority — and every defense is a way to make sure it can’t, including the ones that assume the model will be persuaded and arrange for that to be harmless.

Index S-01 Why agents are differentS-02 Indirect Prompt InjectionS-03 Direct Prompt InjectionS-04 Data ExfiltrationS-05 Confused DeputyS-06 Tool PoisoningS-07 Memory PoisoningS-08 Malicious Retrieved ContentS-09 Credential LeakageS-10 Privilege EscalationS-11 Unsafe Side EffectsS-12 Dual-LLM / Quarantined ReaderS-13 Agent-to-Agent TrustS-14 Sandbox AnatomyS-15 Downstream Sink InjectionS-16 Skill & Plugin Supply ChainS-17 Allow, Ask, Deny: Permission PoliciesS-18 Secrets BrokerS-19 Audit LogsS-20 Denial of Wallet

S-01Why agents are different

/security/chatbot-vs-agent

The same untrusted input, the same model — but one system emits text and the other emits actions.

Step-by-step walkthrough5 steps
  1. Same untrusted input arrives on both sides.
  2. Same class of model processes it.
  3. The chatbot’s blast radius: text a human reads.
  4. The agent’s blast radius: tools with authority.
  5. Side effects are real and often irreversible. Agents differ from chatbots because they possess authority.
Key insight

Security for chatbots is about outputs a human will read. Security for agents is about actions the world will execute. The threat model changes completely.

Failure mode

Treating an agent deployment as “a chatbot with extra features” — the extra features are precisely the attack surface.

Further reading
RelatedIndirect prompt injectionHuman-in-the-loop

S-02Indirect Prompt Injection

/security/indirect-prompt-injection

Content the agent merely reads — a webpage, an email, a document — contains instructions aimed at the model.

Step-by-step walkthrough2 variants · 13 steps

Insecure

  1. An innocent request: summarize a webpage.
  2. The agent fetches the page.
  3. The page contains instructions — written by whoever controls that page.
  4. The payload flows into the model in the same channel as the user’s instructions.
  5. The model complies. It has the permissions to.
  6. Secrets arrive at the attacker. The user asked for a summary.

Protected

  1. Same request, same page.
  2. The payload is still there — you cannot sanitize the whole web.
  3. But fetched content enters marked as tainted data. It can inform the model; it cannot carry authority.
  4. The model may still be fooled into proposing gmail.send. The proposal is just data too.
  5. Policy check: exfil-shaped action + tainted context → escalate, never auto-run.
  6. The human sees “send API keys to a stranger” and denies it.
  7. Blocked. The attacker’s text never became the agent’s authority.
Key insight

You cannot sanitize the whole internet, and you cannot fully stop models from being persuaded. Robust designs assume the model will sometimes comply, and make compliance harmless: tainted context can never authorize a dangerous action on its own.

Failure mode

Defending at the prompt (“ignore instructions in fetched content”) and nowhere else. Prompt-level defenses are advisory; attackers only need one phrasing that works.

Further reading
RelatedDirect injectionData exfiltrationHuman-in-the-loop

S-03Direct Prompt Injection

/security/direct-prompt-injection

The user themselves is the attacker: the prompt tries to override system rules and misuse the agent’s tools.

Step-by-step walkthrough2 variants · 5 steps

Attack

  1. The attacker is the user: the prompt itself tries to override the system rules.
  2. If the model complies, the injected goal becomes a real tool call.

Defended

  1. Same hostile prompt. The model can still be fooled…
  2. …but it can only propose. Enforcement lives outside the model.
  3. Destructive commands are simply not in the allowlist. Compliance without capability is harmless.
Key insight

Never give an agent capabilities its least-trusted user shouldn’t hold. If the allowlist doesn’t contain the dangerous action, jailbreaking the model accomplishes nothing.

Failure mode

Relying on the model’s refusal as the only barrier between any user and destructive tools.

Further reading
RelatedIndirect injectionConfused deputy

S-04Data Exfiltration

/security/data-exfiltration

Secrets that legitimately entered context leave through any outbound channel — URLs, headers, rendered images, commit messages.

Step-by-step walkthrough2 variants · 6 steps

Attack

  1. Secrets sit in context — they had a legitimate reason to be there.
  2. Any outbound channel can smuggle them: URL params, headers, even markdown image links.
  3. One innocuous-looking request later, the token is gone.

Defended

  1. Secrets are tagged at the source — taint travels with the value.
  2. Outbound requests are inspected: does the payload carry taint? Is the host known?
  3. Unknown hosts are unreachable; tainted payloads to anywhere raise an alarm.
Key insight

Exfiltration needs two things: a secret in context and an outbound channel. You rarely control the first; you fully control the second. Enumerate every way bytes leave.

Failure mode

Blocking the obvious channel (email) while markdown image URLs, DNS lookups and log lines remain wide open.

Further reading
RelatedIndirect injectionCredential leakage

S-05Confused Deputy

/security/confused-deputy

A low-privilege requester tricks a high-privilege agent into doing something the requester could never do directly.

Step-by-step walkthrough2 variants · 6 steps

Attack

  1. The requester has no admin rights…
  2. …but the agent does. The API sees a trusted caller and complies.
  3. The agent’s authority was borrowed by someone who never had it.

Defended

  1. Same request arrives.
  2. The agent holds no standing admin power — it borrows the requester’s identity per call.
  3. The API evaluates the real principal and denies. Authority no longer amplifies through the agent.
Key insight

The fix is decades old: don’t act on ambient authority. Every action should be evaluated against the requester’s permissions, not the agent’s.

Failure mode

One service account with admin scope shared across every user of the agent — the classic deputy, now with natural-language persuasion attached.

Further reading
RelatedDirect injectionPrivilege escalation

S-06Tool Poisoning

/security/tool-poisoning

A tool’s own manifest or output carries instructions — and models trust their tools more than the web.

Step-by-step walkthrough2 variants · 7 steps

Attack

  1. A tool is installed from a registry — its manifest is just text.
  2. Hidden instructions ride in the description, loaded straight into context.
  3. Models weight tool descriptions heavily — they’re supposed to follow them.
  4. The poisoned description becomes a real call.

Defended

  1. Same registry — but manifests are pinned to a reviewed hash.
  2. A malicious update is a visible diff, not a silent context change.
  3. The model only ever loads a manifest a human has seen.
Key insight

Tool descriptions are code you execute in the model’s head. Pin them, diff them, review them like dependencies — because that’s what they are.

Failure mode

Auto-updating tool manifests: the version you reviewed is not the version in context.

Further reading
RelatedIndirect prompt injectionMalicious retrieved contentMCP architecture

S-07Memory Poisoning

/security/memory-poisoning

An attacker plants content that gets written to long-term memory — and every future session inherits it as fact.

Step-by-step walkthrough2 variants · 7 steps

Attack

  1. Attacker-controlled content flows through an ordinary session…
  2. …and gets written to long-term memory like any other “fact”.
  3. Weeks later a fresh session recalls it — stripped of where it came from.
  4. One injection, every future session. Memory outlives the attack.

Defended

  1. Writes are gated: unverified claims from untrusted sources rarely qualify.
  2. What does get stored carries its provenance forever.
  3. A later session sees “source: untrusted web” and treats it as a claim, not a fact.
Key insight

Memory is an injection that persists. Provenance is the antidote: a “fact” that remembers it came from an untrusted webpage can be treated accordingly, forever.

Failure mode

Provenance-free stores: once written, attacker text and user truth are indistinguishable.

Further reading
RelatedWrite policies & decayIndirect prompt injection

S-08Malicious Retrieved Content

/security/malicious-retrieved-content

RAG faithfully retrieves whatever ranks well — including documents crafted to rank well and mislead.

Step-by-step walkthrough2 variants · 6 steps

Attack

  1. The retriever ranks by similarity — and this document was built to be similar.
  2. It enters context indistinguishable from your own docs.
  3. Retrieval-augmented misinformation, with citations.

Defended

  1. Retrieval draws from an allowlist; the open web is opt-in, not default.
  2. What passes carries a trust tag the model can reason about.
  3. Low-trust content can inform an answer — it can’t anchor one.
Key insight

Your retriever is an open door with a relevance check, not a truth check. Trust must come from the source list, because it cannot come from the ranking.

Failure mode

Treating “retrieved” as “vetted”: citations make the misinformation more convincing, not less.

Further reading
RelatedRetrieval & RAGTool poisoning

S-09Credential Leakage

/security/credential-leakage

Secrets that legitimately entered context escape through logs, outputs, traces and error messages.

Step-by-step walkthrough2 variants · 6 steps

Attack

  1. The key had a reason to be in context. That’s not the problem.
  2. Everything the model touches is a potential sink: traces, outputs, stack traces.
  3. Logs get shipped, outputs get pasted, errors get screenshotted.

Defended

  1. Known secret values are registered with a redactor at issue time.
  2. Every sink is scrubbed — the literal bytes never leave.
  3. And the key is short-lived anyway: even a miss expires in minutes.
Key insight

Enumerate the sinks, not just the sources: every place model-touched text lands is a place a key can land. Redact at the boundary and make keys short-lived so misses expire.

Failure mode

Long-lived keys plus verbose logging: one debug trace shipped to a vendor is a permanent credential.

Further reading
RelatedData exfiltrationSecrets brokerAudit logsDefense toolbox

S-10Privilege Escalation

/security/privilege-escalation

Individually-safe tools compose into a capability nobody granted.

Step-by-step walkthrough2 variants · 8 steps

Attack

  1. Each tool passed review — individually.
  2. Read the config to find privileged paths…
  3. …write a cron entry there…
  4. …and wait for the scheduler to run it as root.
  5. The composition holds a capability no single tool was granted.

Defended

  1. The read happens — it really is safe alone.
  2. The policy layer sees the chain, not just the call: sensitive read, then a persistence write.
  3. Known escalation compositions are blocked even when every step is allowlisted.
Key insight

Review tool sets, not tools: the question is never “is this call safe?” but “what can the union of these calls build?”

Failure mode

Allowlists that reason per-call: read-config, write-cron and run-script each pass review; together they are root.

Further reading
RelatedConfused deputyDirect prompt injectionPermission policies

S-11Unsafe Side Effects

/security/unsafe-side-effects

No attacker required: write access plus a misunderstanding equals irreversible damage.

Step-by-step walkthrough2 variants · 6 steps

Failure

  1. The agent believes it’s doing what was asked.
  2. Write access plus a misunderstanding is the whole failure.
  3. By the time anyone notices, there is nothing to roll back.

Defended

  1. Destructive intent triggers a dry-run: the exact consequences, before any of them.
  2. A human sees “40,000 rows” — the number the agent’s misunderstanding hid.
  3. Even approved deletes are soft: reversible for 30 days. Irreversibility is a choice, not a default.
Key insight

Design for the honest mistake, not just the adversary — dry-runs, approval on irreversibility, and soft-deletes turn catastrophes into undo operations.

Failure mode

Hard deletes as the default: the cheapest possible implementation of the most expensive possible failure.

Further reading
RelatedHuman-in-the-loopRetry loop

S-12Dual-LLM / Quarantined Reader

/security/dual-llm

The model that holds the tools never reads untrusted text; a quarantined model with no tools reads it and returns only a symbolic handle.

Step-by-step walkthrough8 steps
  1. The user’s task goes to the model that actually holds the credentials.
  2. The task needs a document read — and that document is not trusted.
  3. The privileged model delegates instead of reading. The reader gets no tools and no memory.
  4. The quarantined model fetches the document.
  5. The document carries instructions — and the quarantined model may well comply.
  6. It doesn’t matter. All it can return is a symbolic handle to text it read.
  7. The privileged model plans over $summary without ever seeing what is inside it.
  8. A payload the deciding model never read cannot persuade it. Compliance is made harmless.
Key insight

Every other defense tries to stop the model from being persuaded. This one accepts that it will be, and arranges for the persuaded model to hold no authority — the deciding model plans over handles to text it never saw.

Failure mode

Letting the quarantined output back into the privileged context “just this once”. One inlined summary and the boundary is gone; the whole value was that the text never crossed.

Further reading
RelatedIndirect prompt injectionGuardrail middlewareConfused deputy

S-13Agent-to-Agent Trust

/security/agent-to-agent-trust

One agent’s output is another agent’s untrusted input — and internal provenance is not trust.

Step-by-step walkthrough2 variants · 12 steps

Attack

  1. A read-only research agent fetches a page. Low risk — it holds no dangerous scope.
  2. The page carries instructions aimed at whatever reads it next.
  3. The researcher summarizes. The payload survives the summary, now in the agent’s own voice.
  4. The digest crosses to the implementer — which holds write and deploy scope.
  5. Nothing marks the boundary. Internal provenance reads as trust.
  6. The privileged agent acts on text an attacker wrote. The trust boundary was never drawn.

Defended

  1. Same fetch, same payload. The attack has not changed.
  2. The page still carries instructions. You cannot prevent this half.
  3. The researcher emits structured fields, not prose — and every field records its source.
  4. The implementer’s prompt treats the digest as data to reason about, never as instructions to follow.
  5. Even when persuaded, the implementer can only propose. It does not hold the deploy authority.
  6. The action is authorized against the requesting user’s permissions — not the agent’s.
Key insight

Multi-agent systems quietly launder taint. Text an attacker wrote arrives at the privileged agent in a teammate’s voice, and “it came from our own agent” is exactly the reasoning that makes it dangerous.

Failure mode

Drawing the trust boundary only at the system’s edge. Inside a four-agent pipeline every handoff is an injection point, and the agent holding the write scope is usually the last one.

Further reading
RelatedHierarchicalSingle vs Multi-AgentConfused deputy

S-14Sandbox Anatomy

/security/sandboxing

Disk, process and network are three separate edges — and a sandbox is only as good as the weakest one.

Step-by-step walkthrough7 steps
  1. The agent runs code. The question is never “will it misbehave” but “what can it reach when it does”.
  2. Everything it runs happens inside one box with three edges: disk, process, network.
  3. Disk is a disposable copy. Destroying it costs a rebuild, not a repository.
  4. Host secrets are not mounted — the exfiltration plate’s first ingredient simply is not present.
  5. Every outbound request meets an allowlist, not a firewall rule written after the fact.
  6. Named hosts resolve. The package registry the build genuinely needs still works.
  7. Everything else is unreachable, so an exfiltration channel has nowhere to send to.
Key insight

Sandboxing is the defense that does not depend on the model being right about anything. The question is never whether the agent misbehaves, but what it can reach when it does.

Failure mode

A container with the host’s credentials mounted and unrestricted egress. That is a process boundary, not a security boundary, and it stops nothing that matters.

Further reading
RelatedData exfiltrationUnsafe side effectsSecrets brokerComputer-use coding

S-15Downstream Sink Injection

/security/downstream-sink-injection

An agent writes attacker text into a ticket, PR or doc — and a second agent reads it later as an instruction.

Step-by-step walkthrough2 variants · 12 steps

Attack

  1. Agent A reads untrusted inbound content. So far this is the ordinary injection plate.
  2. The content carries a payload — aimed not at A, but at whatever reads A’s output.
  3. A does its job and files a ticket. The payload rides along inside the description.
  4. The attack goes quiet. It is now sitting in a trusted internal system, indistinguishable from real work.
  5. Hours later a different agent picks the ticket up — and internal tickets are not “untrusted content”.
  6. Agent B executes it. No human was in the loop, and the source is three hops away.

Defended

  1. Same read, same payload. Agent A has no way to tell.
  2. The payload is still there when A writes its ticket.
  3. But the write records provenance: this body was produced by an agent from external content.
  4. Agent B reads it wrapped and quoted — presented as a description, not as a line in its instructions.
  5. Because the input is tainted, B cannot self-authorize. Untrusted provenance downgrades what it may do.
  6. A person sees the request in full before anything is executed.
Key insight

Exfiltration asks how bytes leave. This asks where they land. Any store an agent writes to and another agent reads from is a delayed injection channel, and the delay is what makes it invisible.

Failure mode

Classifying trust by system rather than by provenance. The ticket tracker is internal; the sentence inside the ticket came from the open internet.

Further reading
RelatedIndirect prompt injectionMemory poisoningCI feedback loop

S-16Skill & Plugin Supply Chain

/security/skill-supply-chain

Installing a skill, plugin or subagent definition is executing someone else’s prompt with your own credentials.

Step-by-step walkthrough2 variants · 12 steps

Attack

  1. Skills, plugins and subagent definitions install like packages — from registries anyone can publish to.
  2. What the user reviews is a name, a description and a star rating.
  3. What actually installs is a prompt. Installing an agent definition is executing someone else’s instructions.
  4. It loads above the user’s own text, in the most trusted position in the window.
  5. It inherits every tool the user has already granted. It never needed to ask.
  6. And it runs again every session, long after anyone remembers installing it.

Defended

  1. Same registry, same unreviewed publishers. That half does not change.
  2. A candidate is pinned to an exact version and hash. The thing you reviewed is the thing that loads.
  3. It is read like a dependency: what it instructs, what it can call, what it does on load.
  4. It loads as content the agent reasons about, not as a system instruction it must obey.
  5. And it does not inherit the user’s whole toolbox — a skill gets the scope its job needs.
  6. Its calls still cross the policy layer, so a compromised skill is contained, not catastrophic.
Key insight

A tool manifest is one line of untrusted text; a skill is a whole document that loads above the user’s own instructions and re-runs every session. Review it like a dependency, because that is exactly what it is.

Failure mode

Reviewing the listing instead of the payload. A name, a description and a star rating tell you nothing about the instructions that will load into the most trusted position in the window.

Further reading
RelatedTool poisoningPrivilege escalationGuardrail middleware

S-17Allow, Ask, Deny: Permission Policies

/security/permissions

Per-tool allow, ask and deny rules, scoped by argument, decide each proposed call before it runs; a permission mode sets the default for everything the rules don’t mention.

Step-by-step walkthrough8 steps
  1. The agent proposes a tool call. Nothing runs until a check outside the model answers.
  2. npm test matches an allow rule, so it runs with no prompt — the common case stays fast.
  3. git push matches an ask rule. The same tool, a riskier argument: a person decides.
  4. Approved once — and the approval covers exactly that command, not the whole shell.
  5. Reading a private key matches a deny rule. Deny is checked first; no allow rule can carve an exception.
  6. A call no rule mentions falls through to the session’s permission mode.
  7. Modes run from read-only to autonomous: each step right auto-approves more of what is left unmatched.
  8. Explicit deny rules hold in every mode — autonomy widens the defaults, never the hard limits.
Key insight

Scope rules by argument, not by tool: the same shell that runs the tests can push to main or read a private key. Evaluate deny before allow so a broad block can never be undone by a narrow exception.

Failure mode

Rules so coarse that every call asks. People approve on reflex, then widen the mode to make the prompts stop, and the only rule left standing is “allow everything”.

Further reading
RelatedGuardrail middlewareHuman-in-the-loopPrivilege escalationAudit logs

S-18Secrets Broker

/security/secrets-broker

The agent never holds a raw credential: it requests an action, and a broker at the egress boundary attaches a scoped, short-lived one.

Step-by-step walkthrough2 variants · 11 steps

Key in context

  1. The simplest setup: the key lives in the environment, and the agent reads it like any other value.
  2. The agent attaches it to requests itself. It works — and the model now holds the literal secret.
  3. That key is broad and long-lived, and it lands in every transcript, trace and error the agent produces.
  4. An injected page asks the agent to print its environment.
  5. The key leaves — in a reply, a URL, a commit — and keeps working until someone rotates it.

Brokered

  1. The agent runs with no credential at all — only the ability to ask for an action.
  2. It requests an action, not a key: charge order 91.
  3. The broker checks the action against this run’s scope and fetches a token that covers only it.
  4. The credential is attached at the egress boundary, after the request has left the agent’s reach.
  5. The same injected page asks for the environment — then for refund(all).
  6. There is nothing to print, and an out-of-scope request is refused before any key is attached.
Key insight

Redaction tries to stop a secret from leaving; brokering means it was never there to leave. The model can be persuaded to print anything in its context, so the only safe credential is one outside it.

Failure mode

A broker that hands the token back to the agent “so it can make the call itself”. The key is in context again, and the broker has become a very elaborate environment variable.

Further reading
RelatedCredential leakageSandbox anatomyConfused deputy

S-19Audit Logs

/security/audit-logs

An append-only record of every tool call, its arguments, the approval decision and the actor — which agent, on whose behalf.

Step-by-step walkthrough7 steps
  1. A human, an agent and its tools all act. Every action is also an event.
  2. Each record names the actor — which agent, which subagent, on whose behalf — plus the call and its arguments.
  3. Approval decisions are events too — including every denial: who decided, on exactly what, and when.
  4. Tool results carry secrets — a bearer token in a response header.
  5. Redaction runs before the write, so the log never becomes the next place a key leaks.
  6. Records are appended, chained and never edited — not even by the agent they describe.
  7. That record is what makes forensics, replay of the tool calls and anomaly detection possible.
Key insight

A trace is for debugging a run; an audit log is tamper-evident and names who authorized what. You cannot investigate an incident, replay a bad run’s tool calls or spot a drifting agent from logs that name no actor and drop the denied calls.

Failure mode

Logging the secrets along with everything else. A complete, well-retained log with bearer tokens in it is the most valuable file in the building.

Further reading
RelatedPermission policiesCredential leakageTrajectory evalTracing an agent run

S-20Denial of Wallet

/security/denial-of-wallet

An attacker — or a bug — drives an agent into endless loops, giant contexts or wide fan-out until the money or the shared quota runs out.

Step-by-step walkthrough2 variants · 12 steps

Attack

  1. One request arrives — crafted to be endless, or simply a task the agent cannot finish.
  2. The loop never converges. Nothing in the harness counts the retries.
  3. Every turn re-sends a context that only grows, so each iteration costs more than the last.
  4. It fans out: two hundred subagents, each running the same unbounded loop.
  5. Cost is tokens × turns × agents, and all three are growing at once.
  6. By morning the budget is spent and the shared quota is gone — an outage bought one token at a time.

Defended

  1. The same crafted task arrives. You cannot stop requests from being expensive to answer.
  2. Every step passes a governor before tokens are spent — the harness meters, not the model.
  3. It enforces ceilings per run and per user, so one caller cannot spend everyone’s quota.
  4. Fan-out is capped: the agent asked for two hundred subagents and got four.
  5. The same call with the same arguments comes round a third time. Loop detection trips.
  6. The run stops and says why — worst case is the run budget, not the month’s.
Key insight

An agent’s cost is tokens × turns × agents, and an adversary can push all three from one request. Enforce ceilings per run and per user in the harness, because the model has no idea what anything costs.

Failure mode

An organization-wide spend cap as the only limit. It fires once the month’s budget is gone, and then it takes every other user down with the one runaway run.

Further reading
RelatedBudgets & stopping rulesCost & latency as scoresAudit logs

The defense toolbox

15 PRINCIPLES

🔒  Least privilege

Grant the minimum capability set the task needs, per task, not per agent.

🔒  Scoped credentials

Short-lived, per-request tokens carrying the requester’s identity — never ambient admin.

🔒  Trust boundaries

Mark where untrusted data enters; tainted content informs, it never authorizes.

🔒  Sandboxing

Execution in a container with its own filesystem and no lateral network reach.

🔒  Human approval

Irreversible or high-blast-radius actions pause for a person with full context.

🔒  Policy enforcement

A layer outside the model evaluates every proposed action against explicit rules.

🔒  Permission rules

Allow, ask or deny per tool and per argument — deny checked first, in every permission mode.

🔒  Credential brokering

The agent requests actions; a broker at the egress boundary attaches the key, so no secret enters context.

🔒  Audit logging

Append-only records of every call, argument, approval and actor — with secrets redacted before the write.

🔒  Spend ceilings

Per-run budgets, per-user rate limits, fan-out caps and loop detection turn runaway cost into a stopped run.

🔒  Output filtering

Scan and redact what leaves: secrets, PII, injection payloads echoed onward.

🔒  Taint tracking

Values from untrusted sources carry their provenance through the whole flow.

🔒  Read/write separation

Reading agents cannot write; writing agents consume vetted, structured input.

🔒  Network isolation

Egress allowlists: unknown hosts are simply unreachable.

🔒  Tool allowlists

The dangerous tool the model can’t call is the one attack that can’t work.

Elsewhere in the atlas

Harnesses →Evals →Context →Coding Agents →Tools →Ops →