AI Systems Atlas

P-04Streaming & Cancellation

/ops/streaming

The response arrives as ordered events: text renders as it lands, tool-call arguments accumulate until the call is complete, and a cancel can end it at any event.

Step-by-step walkthrough8 steps
  1. Nothing waits for the whole response. The model emits a stream of small, ordered events.
  2. Text deltas render the moment they arrive: the user reads the first words while the rest is still being written.
  3. A tool call streams too — but its arguments arrive as fragments of JSON.
  4. Acting on a fragment is a bug. Half an argument is not an argument.
  5. So fragments accumulate in a buffer, and the arguments are parsed once, when the call ends.
  6. Only then does the tool run, with a complete, valid input.
  7. The user presses stop mid-stream. The harness aborts the request itself, not just the view.
  8. Cancel is a normal ending: partial text is kept and marked incomplete, and a half-built tool call is discarded, never run.
Key insight

Streaming changes what the user waits for — the first token instead of the last — but tool arguments stream as fragments of JSON that are not valid until the call is complete. Render text eagerly, act on tool calls only when complete, and treat cancel as a normal ending rather than an error.

Failure mode

A stop button that only hides the output. The request keeps generating and the agent loop keeps calling tools for a reader who has already left.

Further reading
RelatedSteering a running agentHuman-in-the-loopRate limits & backpressure

Elsewhere in the atlas

Harnesses →Security →Evals →Context →Coding Agents →Tools →