AI Systems Atlas

Production & Operations

An agent that passes its evals on a laptop is a demo; one that holds up at 3am under real traffic is a system. These plates cover what happens after launch: seeing inside a run, surviving crashes and rate limits, streaming to people who are waiting, shipping changes without breaking what works, and knowing what all of it costs.

Index P-01 Tracing an Agent RunP-02 Durable ExecutionP-03 Rate Limits & BackpressureP-04 Streaming & CancellationP-05 Versioning & RolloutP-06 Cost Monitoring in Production

P-01Tracing an Agent Run

/ops/tracing

One trace per run: every model call, tool call and subagent is a timed span under its parent, so any outcome can be walked back to the step that caused it.

Step-by-step walkthrough7 steps
  1. One run, one trace id. Everything the agent does hangs off this root span.
  2. Every model call is a span on the same clock, so you can see where the time went.
  3. Every tool call is a span too, carrying its arguments and what came back.
  4. A subagent gets its own subtree, linked to the span that spawned it.
  5. Each span records what it would take to replay it: version, tokens, cache, latency, why it stopped.
  6. The run returns a wrong answer. The trace shows why: a fetch deep inside the subagent timed out and was papered over.
  7. The failing trace becomes an eval case, so the fix is tested on every release after this one.
Key insight

The final answer tells you that a run went wrong; the trace tells you where. Record what each span would need to be replayed — prompt version, model id, arguments, tokens — and every failing trace is already most of an eval case.

Failure mode

Logging the final answer, or only the model calls. The bug lives in the tool result nobody recorded, and the run cannot be reproduced to find it.

Further reading
RelatedOffline vs online evalsTrajectory evalSubagent isolationCost monitoring

P-02Durable Execution

/ops/durable-execution

Each workflow step journals its result before the next begins, so a crashed run resumes from the log instead of starting over — and every side effect carries an idempotency key.

Step-by-step walkthrough7 steps
  1. A workflow is a list of steps, and two of them change the world outside the agent.
  2. Each step’s result is journaled to a durable log before the next step starts.
  3. The charge carries an idempotency key built from the run and the step — never from the clock.
  4. The worker dies halfway through step 3. Without a log, the only option is to start again and re-run every step — and without a key, charge the card twice.
  5. A new worker replays the log: steps 1 and 2 return their recorded results without running again.
  6. Crash between the charge and the log write? The retry reuses the key, and the API returns the original charge.
  7. Execution resumes at step 3. The run survives the crash, the card is charged exactly once, and the receipt retries under its own key.
Key insight

Restarting from the top is not recovery for an agent: it repays every token and repeats every side effect. The unit of durability is the step, and the key for a side effect comes from the run and the step, never from the clock.

Failure mode

A retry wrapped around a call that is not idempotent. The charge succeeded, the response was lost, and the customer pays twice to a system that was only trying to be reliable.

Further reading
RelatedLong-running agentCheckpointingRetry loopUnsafe side effects

P-03Rate Limits & Backpressure

/ops/rate-limits

A swarm shares one rate limit, so calls go through a queue and a shared concurrency cap, and a 429 means back off with jitter — not try again now.

Step-by-step walkthrough7 steps
  1. A lead agent fans out to forty subagents, and every one of them calls the same model API.
  2. Unthrottled, they all fire at once. The rate limit belongs to the organization, and the API answers the burst with 429s.
  3. Each waits the same fixed second and retries — together. A synchronized retry is just a second burst.
  4. Instead, calls go into one queue that the whole swarm shares.
  5. A single limiter caps what is in flight and paces admissions to the limits the API reports.
  6. A 429 still happens now and then. It says how long to wait; add jitter so retries spread out.
  7. The retry re-enters the queue, not the API. Load arrives smoothly, and when the queue is full, callers wait instead of piling on.
Key insight

Rate limits belong to the account, not the agent: forty subagents each running a polite retry policy still add up to one burst. Throttle at a single shared point, honor retry-after, and check whether your provider counts cached input against the token limit — caching can raise effective throughput.

Failure mode

Synchronized retries. Every client that got a 429 waits the same fixed second and fires again together, turning one burst into a retry storm that never drains.

Further reading
RelatedParallel swarmRetry loopHierarchical agentCache-aware layoutDenial of wallet

P-04Streaming & Cancellation

/ops/streaming

The response arrives as ordered events: text renders as it lands, tool-call arguments accumulate until the call is complete, and a cancel can end it at any event.

Step-by-step walkthrough8 steps
  1. Nothing waits for the whole response. The model emits a stream of small, ordered events.
  2. Text deltas render the moment they arrive: the user reads the first words while the rest is still being written.
  3. A tool call streams too — but its arguments arrive as fragments of JSON.
  4. Acting on a fragment is a bug. Half an argument is not an argument.
  5. So fragments accumulate in a buffer, and the arguments are parsed once, when the call ends.
  6. Only then does the tool run, with a complete, valid input.
  7. The user presses stop mid-stream. The harness aborts the request itself, not just the view.
  8. Cancel is a normal ending: partial text is kept and marked incomplete, and a half-built tool call is discarded, never run.
Key insight

Streaming changes what the user waits for — the first token instead of the last — but tool arguments stream as fragments of JSON that are not valid until the call is complete. Render text eagerly, act on tool calls only when complete, and treat cancel as a normal ending rather than an error.

Failure mode

A stop button that only hides the output. The request keeps generating and the agent loop keeps calling tools for a reader who has already left.

Further reading
RelatedSteering a running agentHuman-in-the-loopRate limits & backpressure

P-05Versioning & Rollout

/ops/versioned-rollout

Prompt, model and tool schemas ship as one versioned release that must pass an eval gate, then shadow and canary traffic, with the last good release one pointer-flip away.

Step-by-step walkthrough7 steps
  1. Three inputs that change independently decide most of an agent’s behavior: the prompt, the model and the tool schemas.
  2. Pin all three into one immutable release id. The model is a pinned snapshot, never a moving alias.
  3. The release must clear an eval gate first: quality, cost and latency against the version it replaces.
  4. Shadow: it runs on copies of real traffic with its write tools stubbed; answers are compared, never shown.
  5. Canary: 5% of real users, watched against the old release on the same metrics. It widens only while it holds.
  6. Online metrics regress on a slice of traffic the offline suite never covered.
  7. Rollback is a pointer flip to r41, not a hotfix under pressure — because r41 was never edited in place.
Key insight

Prompt, model and tool schemas change independently, so version them together and never edit a release in place. Offline evals decide whether a release may start rolling out; only online metrics on real traffic decide whether it may finish.

Failure mode

Hot-editing the production prompt, or deploying against a moving alias. Behavior shifts with no deploy to blame, and there is nothing to roll back to because the old version no longer exists.

Further reading
RelatedRegression evalsOffline vs online evalsCross-model evalsTracing an agent run

P-06Cost Monitoring in Production

/ops/cost-monitoring

Price every call from its usage, attribute it to a run, a tenant and a release, and watch the distribution and the cache hit rate — because the tail writes the bill.

Step-by-step walkthrough7 steps
  1. Every call reports its usage: uncached input, cache reads, cache writes and output — each at a different price.
  2. Price each call and sum it by run, tagged with the feature, the tenant and the release that spent it.
  3. Look at the distribution, not the average. The typical run costs cents.
  4. The tail is where the money goes: one run cost a thousand times the median, because it never stopped looping.
  5. Cache hit rate collapses after a deploy. Nothing errors; every request just re-reads its whole prompt at full price.
  6. Budgets per run and per tenant turn both into alerts within minutes, not on next month’s invoice.
  7. Every alert points at a run, a release and a trace — the exact step that spent the money.
Key insight

An average cost per run hides everything interesting: a few runaway loops can outspend thousands of normal runs. Attribute cost where it is spent, alert on per-run and per-tenant budgets, and treat a falling cache hit rate as a cost regression even when nothing errors.

Failure mode

Finding out from the invoice. The monthly total arrives weeks after the loop that caused it, with no link back to the run, the prompt or the release that spent the money.

Further reading
RelatedBudgets & stopping rulesCost & latency as scoresCache-aware layoutTracing an agent runDenial of wallet

Elsewhere in the atlas

Harnesses →Security →Evals →Context →Coding Agents →Tools →