E-13The Eval Environment

/evals/eval-environment

Each task instance is built fresh from pinned dependencies, seeded data, a frozen clock and recorded network — then destroyed.

Key insight

Reproducibility is a property of the environment, not the metric. If a March result cannot be rebuilt in November, you do not have a benchmark; you have an anecdote with a number attached.

Failure mode

Tasks that share state. One run leaves a file, a row or a cached token behind, and every later run is measuring something nobody designed.

E-13The Eval Environment

Elsewhere in the atlas