AI Systems Atlas

E-11Variance & Sample Size

/evals/variance-and-sample-size

Agents are stochastic, so a single run is one bit — and a one-task difference is usually noise.

Step-by-step walkthrough7 steps
  1. One task, one agent, nothing changed between attempts.
  2. Run it once and you learn one bit: it passed, or it didn’t.
  3. Run the identical input again and the answer can flip. Sampling is not a bug you can prompt away.
  4. Five runs, three passes. That is not 60% — it is 60% with an interval wide enough to drive through.
  5. Now version 2 scores 4 out of 5. One extra pass.
  6. That is not an improvement. It is one coin flip inside the noise you already measured.
  7. Report the interval, not the point. The number of runs is a design decision — pick it before you look.
Key insight

Every other eval plate produces a number; this one tells you whether the number means anything. Decide how many runs per task before you look at the result, and report the interval rather than the point.

Failure mode

Shipping on a delta smaller than the run-to-run spread. You will chase phantom regressions for a week and, worse, ship real ones that happened to land inside the noise.

Further reading
RelatedRegression evalsOutcome evalCross-model evalspass@k vs pass^k

Elsewhere in the atlas

Harnesses →Security →Context →Coding Agents →Tools →Ops →