E-09Cross-Model Evals

Same tasks, same harness, models swapped — so differences are attributable.

Step-by-step walkthrough4 steps

Only one variable changes: the model. Harness, tools and tasks stay fixed.
Each model runs the full set.
Now differences are attributable: “the model got smarter” vs “the harness got better”.
If a new model doesn’t move your eval, your harness is the bottleneck.

Key insight

This is how you separate “the model got smarter” from “the harness got better”, and how you know when a new model release should change your stack.

Failure mode

Prompts silently tuned to one model’s quirks: the comparison measures prompt fit, not model capability.

Elsewhere in the atlas