AI Systems Atlas

P-03Rate Limits & Backpressure

/ops/rate-limits

A swarm shares one rate limit, so calls go through a queue and a shared concurrency cap, and a 429 means back off with jitter — not try again now.

Step-by-step walkthrough7 steps
  1. A lead agent fans out to forty subagents, and every one of them calls the same model API.
  2. Unthrottled, they all fire at once. The rate limit belongs to the organization, and the API answers the burst with 429s.
  3. Each waits the same fixed second and retries — together. A synchronized retry is just a second burst.
  4. Instead, calls go into one queue that the whole swarm shares.
  5. A single limiter caps what is in flight and paces admissions to the limits the API reports.
  6. A 429 still happens now and then. It says how long to wait; add jitter so retries spread out.
  7. The retry re-enters the queue, not the API. Load arrives smoothly, and when the queue is full, callers wait instead of piling on.
Key insight

Rate limits belong to the account, not the agent: forty subagents each running a polite retry policy still add up to one burst. Throttle at a single shared point, honor retry-after, and check whether your provider counts cached input against the token limit — caching can raise effective throughput.

Failure mode

Synchronized retries. Every client that got a 429 waits the same fixed second and fires again together, turning one burst into a retry storm that never drains.

Further reading
RelatedParallel swarmRetry loopHierarchical agentCache-aware layoutDenial of wallet

Elsewhere in the atlas

Harnesses →Security →Evals →Context →Coding Agents →Tools →