AI Systems Atlas

P-02Durable Execution

/ops/durable-execution

Each workflow step journals its result before the next begins, so a crashed run resumes from the log instead of starting over — and every side effect carries an idempotency key.

Step-by-step walkthrough7 steps
  1. A workflow is a list of steps, and two of them change the world outside the agent.
  2. Each step’s result is journaled to a durable log before the next step starts.
  3. The charge carries an idempotency key built from the run and the step — never from the clock.
  4. The worker dies halfway through step 3. Without a log, the only option is to start again and re-run every step — and without a key, charge the card twice.
  5. A new worker replays the log: steps 1 and 2 return their recorded results without running again.
  6. Crash between the charge and the log write? The retry reuses the key, and the API returns the original charge.
  7. Execution resumes at step 3. The run survives the crash, the card is charged exactly once, and the receipt retries under its own key.
Key insight

Restarting from the top is not recovery for an agent: it repays every token and repeats every side effect. The unit of durability is the step, and the key for a side effect comes from the run and the step, never from the clock.

Failure mode

A retry wrapped around a call that is not idempotent. The charge succeeded, the response was lost, and the customer pays twice to a system that was only trying to be reliable.

Further reading
RelatedLong-running agentCheckpointingRetry loopUnsafe side effects

Elsewhere in the atlas

Harnesses →Security →Evals →Context →Coding Agents →Tools →