AI Systems Atlas

E-11Variance & Sample Size

/evals/variance-and-sample-size

Agents are stochastic, so a single run is one bit — and a one-task difference is usually noise.

Key insight

Every other eval plate produces a number; this one tells you whether the number means anything. Decide how many runs per task before you look at the result, and report the interval rather than the point.

Failure mode

Shipping on a delta smaller than the run-to-run spread. You will chase phantom regressions for a week and, worse, ship real ones that happened to land inside the noise.

Further reading
RelatedRegression evalsOutcome evalCross-model evals

Elsewhere in the atlas

Harnesses →Security →Context →Coding Agents →