AI Systems Atlas

S-08Malicious Retrieved Content

/security/malicious-retrieved-content

RAG faithfully retrieves whatever ranks well — including documents crafted to rank well and mislead.

Step-by-step walkthrough2 variants · 6 steps

Attack

  1. The retriever ranks by similarity — and this document was built to be similar.
  2. It enters context indistinguishable from your own docs.
  3. Retrieval-augmented misinformation, with citations.

Defended

  1. Retrieval draws from an allowlist; the open web is opt-in, not default.
  2. What passes carries a trust tag the model can reason about.
  3. Low-trust content can inform an answer — it can’t anchor one.
Key insight

Your retriever is an open door with a relevance check, not a truth check. Trust must come from the source list, because it cannot come from the ranking.

Failure mode

Treating “retrieved” as “vetted”: citations make the misinformation more convincing, not less.

Further reading
RelatedRetrieval & RAGTool poisoning

Elsewhere in the atlas

Harnesses →Evals →Context →Coding Agents →Tools →Ops →