P-06Cost Monitoring in Production

/ops/cost-monitoring

Price every call from its usage, attribute it to a run, a tenant and a release, and watch the distribution and the cache hit rate — because the tail writes the bill.

Step-by-step walkthrough7 steps

Every call reports its usage: uncached input, cache reads, cache writes and output — each at a different price.
Price each call and sum it by run, tagged with the feature, the tenant and the release that spent it.
Look at the distribution, not the average. The typical run costs cents.
The tail is where the money goes: one run cost a thousand times the median, because it never stopped looping.
Cache hit rate collapses after a deploy. Nothing errors; every request just re-reads its whole prompt at full price.
Budgets per run and per tenant turn both into alerts within minutes, not on next month’s invoice.
Every alert points at a run, a release and a trace — the exact step that spent the money.

Key insight

An average cost per run hides everything interesting: a few runaway loops can outspend thousands of normal runs. Attribute cost where it is spent, alert on per-run and per-tenant budgets, and treat a falling cache hit rate as a cost regression even when nothing errors.

Failure mode

Finding out from the invoice. The monthly total arrives weeks after the loop that caused it, with no link back to the run, the prompt or the release that spent the money.

P-06Cost Monitoring in Production

Elsewhere in the atlas