AI Systems Atlas

E-12Cost & Latency as Scores

/evals/cost-and-latency

Pass rate is one axis; tokens and seconds per task are the other two, and they move in the opposite direction.

Step-by-step walkthrough6 steps
  1. Two versions of an agent over the same benchmark.
  2. Pass rate is up four points. On most dashboards this is where the story ends.
  3. It burns three times the tokens to get there.
  4. And it takes more than twice as long, which users feel on every single request.
  5. Four points for 3× the spend is a trade, and it is one a person has to make deliberately.
  6. Score all three every run. An eval that only measures quality cannot tell you the price of it.
Key insight

Almost every technique in this atlas buys quality with compute. If your eval only reports quality, every one of them looks free, and the bill arrives in production.

Failure mode

A leaderboard with one column. Four points of pass rate for three times the spend may be a great trade or a terrible one — but nobody can tell from a single number.

Further reading
RelatedRegression evalsBest-of-NRouter / cascadeCost monitoring

Elsewhere in the atlas

Harnesses →Security →Context →Coding Agents →Tools →Ops →