AI Systems Atlas

E-01Outcome Eval

/evals/outcome-eval

Run the task end to end; check only the final result.

Step-by-step walkthrough3 steps

Run the task end to end.
Only the final artifact is inspected — the trajectory is a black box.
Binary verdict. Cheap, objective, and blind to how.

Key insight

Ground truth where you can get it — a passing test, a correct SQL result, a booked meeting — is worth a thousand vibes.

Failure mode

Outcome-only metrics reward lucky trajectories and hide near-misses; they tell you that it failed, never why.

Further reading

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Jimenez et al., ICLR 2024
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains Yao et al., 2024

Elsewhere in the atlas