AI Systems Atlas

E-12Cost & Latency as Scores

/evals/cost-and-latency

Pass rate is one axis; tokens and seconds per task are the other two, and they move in the opposite direction.

Key insight

Almost every technique in this atlas buys quality with compute. If your eval only reports quality, every one of them looks free, and the bill arrives in production.

Failure mode

A leaderboard with one column. Four points of pass rate for three times the spend may be a great trade or a terrible one — but nobody can tell from a single number.

Further reading
RelatedRegression evalsBest-of-NRouter / cascade

Elsewhere in the atlas

Harnesses →Security →Context →Coding Agents →