E-10Offline vs Online Evals

Static benchmarks are controlled but stale; production monitoring is live but noisy. Run both.

Key insight

The gap between offline and online numbers is itself the metric: it measures how far your benchmark has drifted from reality.

Failure mode

Trusting the offline number when they disagree. Production is noisy, but it is not wrong.

Elsewhere in the atlas