G-13The Verifier Ladder
Format, lint, typecheck, unit, integration, CI — ordered by seconds to signal, cheapest first.
Further reading
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering Yang et al., NeurIPS 2024
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Jimenez et al., ICLR 2024
- Training Verifiers to Solve Math Word Problems Cobbe et al., OpenAI 2021