AI Systems Atlas

G-20Reviewing Agent-authored PRs

/coding-agents/reviewing-agent-prs

What a reviewer, human or bot, checks in an agent’s pull request that green CI does not: scope, tests, real APIs and evidence.

Step-by-step walkthrough9 steps
  1. Agents open pull requests faster than any team has ever received them.
  2. Each arrives green, with a summary and the trajectory: every command the agent ran and what came back.
  3. They all converge on one scarce resource — a person’s attention. Review, not generation, is now the bottleneck.
  4. Scope creep: files the task never mentioned, “tidied” along the way. Green CI does not care.
  5. Existing tests weakened in the same diff — an assertion loosened until it passed.
  6. An API that does not exist, called confidently, behind a mock that makes the test pass.
  7. The trajectory is the evidence: did it run the tests and read the output, or only say it did?
  8. Any “no” goes back with its reason, as a narrower task — not quietly fixed by the reviewer.
  9. Four yeses → merge. Small, scoped PRs with evidence attached are what keep each review cheap.
Key insight

Once agents write most of the diffs, review is the bottleneck, so the design goal is a cheap review: small scoped PRs, a summary with evidence attached, and a reviewer bot for the first pass. The trajectory is the cheapest evidence there is.

Failure mode

Reviewing an agent PR like a trusted colleague’s. A confident, tidy, green diff invites a skim, and a skim misses the loosened assertion and the helper that doesn’t exist.

Further reading
RelatedReviewer agentBackground agentsTest-driven agentTrajectory eval

Elsewhere in the atlas

Harnesses →Security →Evals →Context →Tools →Ops →