AI Systems Atlas

H-22Thinking Budgets

/harnesses/thinking-budgets

The model spends reasoning tokens before it answers, and the harness sets how much it may spend — a token budget or an effort level — trading latency and cost for accuracy.

Step-by-step walkthrough7 steps
  1. A prompt arrives. Before answering, the model can spend tokens reasoning.
  2. The harness sets how many. A low budget is fast and cheap…
  3. …and a multi-step problem gets a shallow pass. The answer comes back quickly, and wrong.
  4. A high budget thinks longer: accuracy goes up, and so do latency and cost.
  5. Returns diminish: accuracy grows with the log of the budget, so each doubling costs twice as much for about the same gain.
  6. Adaptive effort: judge difficulty per request — with a classifier in the harness, or by letting the model decide within an effort level.
  7. Easy requests get little or no thinking; hard ones get room. The spend lands where it changes the answer.
Key insight

Accuracy tends to climb with thinking tokens but with diminishing returns, so the right budget depends on the request. Adaptive effort — deciding per request how hard to think — usually beats any single fixed setting.

Failure mode

One budget for everything. A high fixed budget pays hard-problem prices on easy requests; a low one quietly degrades exactly the requests that needed the thinking.

Further reading
RelatedBudgets & stopping rulesRouter / cascadeInterleaved thinking

Elsewhere in the atlas

Security →Evals →Context →Coding Agents →Tools →Ops →