AI Systems Atlas

P-05Versioning & Rollout

/ops/versioned-rollout

Prompt, model and tool schemas ship as one versioned release that must pass an eval gate, then shadow and canary traffic, with the last good release one pointer-flip away.

Step-by-step walkthrough7 steps
  1. Three inputs that change independently decide most of an agent’s behavior: the prompt, the model and the tool schemas.
  2. Pin all three into one immutable release id. The model is a pinned snapshot, never a moving alias.
  3. The release must clear an eval gate first: quality, cost and latency against the version it replaces.
  4. Shadow: it runs on copies of real traffic with its write tools stubbed; answers are compared, never shown.
  5. Canary: 5% of real users, watched against the old release on the same metrics. It widens only while it holds.
  6. Online metrics regress on a slice of traffic the offline suite never covered.
  7. Rollback is a pointer flip to r41, not a hotfix under pressure — because r41 was never edited in place.
Key insight

Prompt, model and tool schemas change independently, so version them together and never edit a release in place. Offline evals decide whether a release may start rolling out; only online metrics on real traffic decide whether it may finish.

Failure mode

Hot-editing the production prompt, or deploying against a moving alias. Behavior shifts with no deploy to blame, and there is nothing to roll back to because the old version no longer exists.

Further reading
RelatedRegression evalsOffline vs online evalsCross-model evalsTracing an agent run

Elsewhere in the atlas

Harnesses →Security →Evals →Context →Coding Agents →Tools →