AI Systems Atlas

E-18Benchmark Saturation

/evals/benchmark-saturation

Once scores bunch against the ceiling, the gaps between models shrink below the noise and the benchmark stops ranking anything.

Step-by-step walkthrough6 steps
  1. A new benchmark launches. The best model solves 64% of it.
  2. Scores spread over 30 points: the ranking is obvious, and a 5-point gain is a real result.
  3. Two years of everyone hill-climbing the same fixed set…
  4. …and every model lands within two points of the ceiling — a ceiling set by mislabeled and unsolvable tasks, not by capability.
  5. The gaps are now smaller than run-to-run noise. The benchmark still produces numbers; it no longer ranks anything.
  6. Retire or refresh it: a harder split, fresh tasks, a private set — until the scores spread out again.
Key insight

The ceiling is rarely 100%: mislabeled and unsolvable tasks cap it lower, so the last few points measure grader errors, not capability. Watch the spread of top scores, and refresh with harder splits, fresh tasks or a private set before it collapses.

Failure mode

Celebrating a 1-point lead on a saturated benchmark. It is inside the run-to-run noise, and on the tasks that remain it may be the model memorizing a wrong answer key.

Further reading
RelatedTask sets & contaminationVariance & sample sizeRegression evals

Elsewhere in the atlas

Harnesses →Security →Context →Coding Agents →Tools →Ops →