AI Systems Atlas

E-16pass@k vs pass^k

/evals/pass-at-k-vs-pass-hat-k

pass@k counts a task solved if any of k tries succeeds; pass^k only if all k do — capability versus reliability.

Step-by-step walkthrough8 steps
  1. One task, one agent that succeeds 75% of the time on any single try.
  2. Run it k = 3 times. Two passes, one failure — nothing about the agent changed.
  3. pass@3 asks: did any try pass? Yes — the task counts as solved.
  4. pass^3 asks: did every try pass? One failure sinks it — the task counts as failed.
  5. At k = 1 the two metrics are the same number.
  6. At k = 3 they have split: 98% vs 42%, from one unchanged agent.
  7. At k = 8, pass@k says solved; pass^k says all eight runs succeed only one time in ten.
  8. Pick the one your product lives by: best-of-k behind a verifier wants pass@k; an agent every customer runs wants pass^k.
Key insight

With independent tries at 75% each, three tries give 98% pass@3 and 42% pass^3. The same agent looks nearly perfect or unshippable depending on which question you ask, so ask the one your product actually poses.

Failure mode

Reporting pass@k for a customer-facing agent. Users do not get k tries and a verifier — each of them gets one run, and they come back tomorrow for another.

Further reading
RelatedVariance & sample sizeBest-of-NOutcome eval

Elsewhere in the atlas

Harnesses →Security →Context →Coding Agents →Tools →Ops →