AI Systems Atlas

P-01Tracing an Agent Run

/ops/tracing

One trace per run: every model call, tool call and subagent is a timed span under its parent, so any outcome can be walked back to the step that caused it.

Step-by-step walkthrough7 steps
  1. One run, one trace id. Everything the agent does hangs off this root span.
  2. Every model call is a span on the same clock, so you can see where the time went.
  3. Every tool call is a span too, carrying its arguments and what came back.
  4. A subagent gets its own subtree, linked to the span that spawned it.
  5. Each span records what it would take to replay it: version, tokens, cache, latency, why it stopped.
  6. The run returns a wrong answer. The trace shows why: a fetch deep inside the subagent timed out and was papered over.
  7. The failing trace becomes an eval case, so the fix is tested on every release after this one.
Key insight

The final answer tells you that a run went wrong; the trace tells you where. Record what each span would need to be replayed — prompt version, model id, arguments, tokens — and every failing trace is already most of an eval case.

Failure mode

Logging the final answer, or only the model calls. The bug lives in the tool result nobody recorded, and the run cannot be reproduced to find it.

Further reading
RelatedOffline vs online evalsTrajectory evalSubagent isolationCost monitoring

Elsewhere in the atlas

Harnesses →Security →Evals →Context →Coding Agents →Tools →