AI Systems Atlas

G-13The Verifier Ladder

/coding-agents/verifier-ladder

Format, lint, typecheck, unit, integration, CI — ordered by seconds to signal, cheapest first.

Step-by-step walkthrough7 steps
  1. The agent makes an edit. Something has to tell it whether that edit was any good.
  2. Not one check — a ladder, ordered by how fast each rung can answer.
  3. The cheap rungs run first and catch the most common mistakes in seconds.
  4. A type error at rung three — and the ladder stops. The four-minute suite is never started.
  5. Failure goes straight back to the edit, with the traceback, seconds after the mistake.
  6. Fix and climb again. Most iterations never leave the first three rungs.
  7. Only a change that has cleared every cheap rung is allowed to spend twenty-two minutes of CI.
Key insight

The test loop draws one verifier; in practice it is a ladder, and the ordering is the design. Most iterations never leave the bottom three rungs, which is what makes an agent that edits fifty times an hour affordable.

Failure mode

One monolithic “run the tests” step. The agent waits four minutes to be told about a syntax error, and every retry costs the full suite again.

Further reading
RelatedThe test / diagnose loopCI feedback loopActor–VerifierLanguage-server integration

Elsewhere in the atlas

Harnesses →Security →Evals →Context →Tools →Ops →