AI Systems Atlas

X-11Cache-aware Context Layout

/context/cache-aware-layout

Order the window by how often each part changes, because a prompt cache is a prefix match and the first mutation invalidates everything after it.

Step-by-step walkthrough7 steps
  1. The window is not a bag of content. It is an ordered sequence, and order is the whole lever.
  2. Sort it by how often each part changes: never, rarely, every turn.
  3. Only the tail moves between turns.
  4. So the long, expensive prefix is served from cache — a large fraction of the tokens, at a fraction of the price and the latency.
  5. Now put a timestamp in the system prompt, or let the tool list come back in a different order.
  6. Everything after that byte is re-read. The cache is a prefix match, not a similarity match.
  7. This is why compaction is expensive twice: it rewrites the middle, so the whole tail after it pays again.
Key insight

This reframes half of this collection: compaction, sliding windows and memory loading are not only token decisions, they are cache decisions. A rewrite in the middle makes the whole tail after it expensive again.

Failure mode

A timestamp, a session id or an unsorted tool list in the system prompt. One volatile byte at the front and the cache never hits — for every request, forever.

Further reading
RelatedAnatomy of a context windowContext compactionSliding windowCost monitoring

Elsewhere in the atlas

Harnesses →Security →Evals →Coding Agents →Tools →Ops →