E-10Offline vs Online Evals

Static benchmarks are controlled but stale; production monitoring is live but noisy. Run both.

Step-by-step walkthrough4 steps

Offline: same tasks every run. Perfect for regressions, blind to drift.
Online: sampled real traffic, judged continuously. Noisy, but true.
94% vs 81%. When they disagree, production is right.
Feed the online failures back into the offline set. That loop is the eval program.

Key insight

The gap between offline and online numbers is itself the metric: it measures how far your benchmark has drifted from reality.

Failure mode

Trusting the offline number when they disagree. Production is noisy, but it is not wrong.

Elsewhere in the atlas