A cheaper model is not automatically a cheaper run

Routing a request to a cheaper model saves money only if the run finishes for less. Two things sit between the per-token price and the run total: the switch usually lands on a model whose prompt cache is cold, and a cheaper model that gets the task wrong is retried or escalated. A savings figure that accounts for neither is comparing list prices, not runs.

A savings number is a claim about a run that never happened

Every "the router cut costs by N%" figure answers a counterfactual: what would this traffic have cost on the model you would otherwise have used? Nothing records that model, because the request never went to it. So the baseline is chosen, and the choice sets the number.

LiteLLM's maintainers are working through this in the open, which makes it a useful place to watch the problem get specified. A pull request adding auto-router savings to their cost dashboard states the gap plainly:

Dashboard credits compression and prompt caching, not auto-router model switches

Its stated method is to price "the traffic against the priciest model the router could pick" — a defensible convention, and one worth naming out loud, because the same traffic scored against the model a human would actually have chosen produces a smaller number. Neither is wrong. They are different questions, and a percentage on its own does not say which one was asked.

Worth being fair about the tool: LiteLLM is MIT-licensed and free, covers 140+ providers, and enforces hard budgets per key, team, organisation and model. If the job is multi-provider routing at the edge, that is the thing to reach for. The accounting question here is not a criticism of it — it is a question every routing layer inherits, including ours.

The switch itself pays a cache write

Provider prompt caches are per model. Move a session from one tier to another and the prefix that was being read at a discount on the old model does not exist on the new one. A LiteLLM pull request on cache warming describes the mechanism and its cost:

provider caches are per-model and expire in about five minutes, so a session that moves between tiers pays a full cache write every time it lands on a model whose cache has gone cold, losing the ~90 percent cache read discount

That is the direction Anthropic's pricing points too: cache reads cost about a tenth of base input and writes about 1.25×, so a stable prefix pays for itself on first reuse — and a switch resets that clock. On a long session with a large system prompt, the write can be a material share of what the cheaper tier was supposed to save. If the router switches back a few turns later, it pays again.

The obvious fix has its own cost. The same pull request notes that pinning a session to one model avoids the repeated writes, but "the pin means a session that starts simple never reaches a stronger model" — the escalation the router existed for.

There is also a measurement trap in the raw signal. A later PR puts it as: "A cold cache means two opposite things and they need opposite math." A cold cache on the first turn of a conversation is normal and nobody caused it. A cold cache on turn nine was caused by the switch, and charging both the same way flatters or punishes the router at random. If you are auditing your own numbers, that distinction is the first thing to check.

The prerequisite for any of this is a prefix that could have been cached in the first place. A prompt assembled with an f-string at the top never had a reusable prefix, so no routing policy will change its cache behaviour — see prompt prefix stability and why caching silently stops working.

The reason a request landed on a tier has to be on the record

Post-hoc savings maths is only auditable if each event says which rule fired. Another LiteLLM PR describes the state before that existed: an auto-routed request's log recorded the model group and the resolved deployment, "but nothing about which tier was chosen or what chose it, so 'why did this prompt land on the cheap model' is unanswerable from the logs" (#35016).

Agent traffic makes the classification harder than chat traffic. A fix to the same router's classifier reports that it only saw the last user message, and that mid-loop the last message is often harness boilerplate rather than the request — so a hard follow-up gets scored on text the user never wrote. Anything that decides model tier from the tail of an agent's message list is deciding on the wrong input a lot of the time.

Published percentages do not transfer

A pull request proposing a new router provider for CrewAI lists "70-95% cost savings" versus single-model setups in its motivation. An engineering team at Sword Health published "We built an AI model router that cut LLM costs by 94%" the same week. Both are worth reading. Neither is a number you can adopt, because the savings depend on your traffic mix, how much of your input was cacheable, and how often the cheap tier's output has to be redone.

What to measure on your own traffic

Four quantities, all of which need per-call records you control:

1. Cost per completed run, before and after the rule. Not cost per call and not a blended per-token rate — a router that halves the price per call and adds a retry has not saved anything. Cost per run is the unit the counterfactual has to be expressed in.
2. Cache-read share of input tokens, per model, after a switch. If it collapses when the router moves a session, the write is real and it belongs in the savings maths.
3. Retry and escalation rate on the cheap tier. A run that fails there and reruns on the expensive model costs both, plus the operator time.
4. Which rule fired, recorded on the event. Without it, every savings figure is unfalsifiable.

How this works in Capsera

Routing runs in the SDK, before the provider call. Four rule types are evaluated in order — cost cap, task type, model map, then fallback chain — the first match wins, cross-provider routing is blocked so a rule cannot quietly change vendors, and the evaluation is fail-open: if rules cannot be fetched, the call proceeds on the model your code asked for.

What that buys on the measurement side is the unit. Because every call carries an agent identity and a run identifier, the comparison available is cost per run for the same agent before and after a rule, rather than a per-token average across everything. Cache-read share comes from the same events, so the cache write a switch caused shows up next to the switch that caused it rather than in a separate ledger.

The part we will not do for you is pick the baseline. That is a judgement about what your system would have done otherwise, and it belongs to whoever has to defend the number.