AI Systems Atlas

H-16Router / Model Cascade

/harnesses/model-router

A cheap triage step picks the model; a quality check escalates only what needs escalating.

Step-by-step walkthrough8 steps
  1. A request arrives. Nothing about it says which model it needs.
  2. A cheap triage step routes it — a classifier or a very small model, never the expensive one.
  3. Most requests go to the small model first.
  4. A check decides whether the draft is actually good enough to ship.
  5. Good enough → done, at a fraction of the cost and the latency.
  6. Not good enough → escalate. The cheap attempt was still cheap.
  7. Triage can also skip the ladder outright when a request is obviously hard.
  8. One answer either way — the caller never learns which model served it.
Key insight

Most traffic is easy. Routing lets you price for the median request instead of the worst one, and a failed cheap attempt is still cheap.

Failure mode

A router that costs as much as what it routes to. If triage needs the big model to decide, the cascade has bought latency and nothing else.

Further reading
RelatedBest-of-NSingle-shotCross-model evalsThinking budgets

Elsewhere in the atlas

Security →Evals →Context →Coding Agents →Tools →Ops →