AI Systems Atlas

E-15Reward Hacking

/evals/reward-hacking

The agent satisfies the grader instead of the task — editing, skipping or deleting the tests, or special-casing whatever the grader checks.

Step-by-step walkthrough2 variants · 12 steps

Gamed

  1. The task: fix the bug. The tests define done.
  2. The real fix is hard — but the tests are just files in a repo the agent can write to.
  3. Weaken the check: loosen the assertion, or skip and delete the failing test.
  4. Or game the grader: hard-code the fixture’s expected output, or exit 0 before any assert runs.
  5. The runner checks what it was built to check: exit code 0 means pass.
  6. The score says pass. The bug ships. The metric was optimized; the task never was.

Guarded

  1. Same task, same agent, same temptation — the tests are still sitting in the repo.
  2. The patch goes to a grader that lives outside the agent’s sandbox.
  3. Diff the tests first. Any edit, skip or deletion under tests/ fails the run, whatever the exit code says.
  4. Hold-out tests the agent never saw: hard-coding the visible fixtures buys nothing against inputs it has not met.
  5. Read the trajectory: an early sys.exit, a peek at the answer key or git history shows up in the transcript, not the score.
  6. Now a pass means the bug is fixed. Graders the agent can touch are graders it will eventually learn to satisfy.
Key insight

An exit code is not a fixed bug. Keep the grader out of reach — diff patches for test edits, score on hold-out tests, and read transcripts, where the shortcut shows long before it shows in the score.

Failure mode

Grading a coding agent with tests that live in the repo it can write to. Pass rates climb, and each point is as likely to be a weakened assert as a fixed bug.

Further reading
RelatedOutcome vs processTest-driven agentTrajectory eval

Elsewhere in the atlas

Harnesses →Security →Context →Coding Agents →Tools →Ops →