AI Systems Atlas

E-13The Eval Environment

/evals/eval-environment

Each task instance is built fresh from pinned dependencies, seeded data, a frozen clock and recorded network — then destroyed.

Step-by-step walkthrough7 steps
  1. A task is a spec: how to set the world up, and what must be true at the end.
  2. Every run builds its own instance. Nothing is shared between runs or between tasks.
  3. Dependencies pinned, data seeded — so a result from March still reproduces in November.
  4. Clock frozen and network recorded: the two things that silently make a benchmark irreproducible.
  5. The agent gets exactly this world and no other.
  6. The assertion runs inside the instance, against real final state rather than the agent’s account of it.
  7. Then it is destroyed. A benchmark you cannot rebuild is a number you cannot defend.
Key insight

Reproducibility is a property of the environment, not the metric. If a March result cannot be rebuilt in November, you do not have a benchmark; you have an anecdote with a number attached.

Failure mode

Tasks that share state. One run leaves a file, a row or a cached token behind, and every later run is measuring something nobody designed.

Further reading
RelatedOutcome evalVariance & sample sizeEnvironment bootstrap

Elsewhere in the atlas

Harnesses →Security →Context →Coding Agents →Tools →Ops →