The experiment that measures a saving has its own bill

A model comparison spends money on every arm it runs. The candidate model, the arm it is measured against, and — where the scoring is done by another model — the judge all bill at production rates, so the saving the experiment reports is gross of what the experiment cost to produce. Five items from the past week, across three projects, each land on one of those arms being unpriced, unbudgeted or unattributed.

A ceiling in turns does not bound what a comparison spends

A LiteLLM pull request from 2026-08-20 gates shadow-eval budgets on dollar spend instead of turns and states the problem in two lines:

Shadow eval budgets are turn counts, but the thing operators care about is dollars

A turn's cost varies wildly by model and prompt, so max_turns bounds spend only loosely

The fix replaces max_turns with a max_budget in USD per key. A turn cap is a weak proxy for spend in any workload — that argument generalises to run quotas and concurrency caps — but an experiment is the case where it is weakest by construction. The whole point of the run is that one arm uses a model you have not been billed for yet, at a prompt size you have not measured yet. A count that was calibrated against the incumbent is calibrated against the wrong arm.

The judge is the arm most likely to be priced at nothing

The same pull request reports what its own accounting did:

The job's own judge_spend read $0.00 whenever name-based pricing missed

A judge model is chosen inside an evaluation config rather than in the application, and it is often not the model the rest of the system runs on. Its price is then looked up by a name that nobody has had a reason to add to the table. A missed lookup returns zero rather than an error, so the measurement arm drops out of the total in the direction that flatters the result: the experiment looks free and the saving looks clean. This is the same failure surface as a model having no single price per token, reached from a different side — there the rate was wrong, here there is no row at all.

Each arm needs an identity before the run, not a query after it

Two more LiteLLM changes the same week are about keeping arms separable. One that scopes shadow eval jobs to multiple keys, from 2026-08-18, gives each arm its own state:

Each key keeps its own turn budget and stop state

A second, from the same day, adds a per-key breakdown to the dashboard because the numbers existed and could not be read:

Per key budgets and win rates are invisible on screen

A win rate without the spend that produced it is half a comparison. The reason both halves have to be designed in is what a question on the Langfuse discussions board from 2026-08-21 makes plain:

I wish to have a dashboard to monitor the cost related to llm as a judge evaluator, splitting by experiment run.

Whether that is answerable depends entirely on whether the experiment identifier was attached to the call when the call was made. Split-by-experiment is a group by, and a group by needs the column. Asked afterwards, it becomes an attempt to reconstruct which arm a request belonged to from timestamps and model names, which is exactly the join that is not recoverable once the rows are written.

A measurement path bills at production rates

None of this is specific to shadow testing. Any path that exists to check something — a regression suite, a bake-off, a nightly comparison — makes real provider calls that arrive on the same invoice as the product. We scanned 133 public Python agent repositories; 51 made direct LLM calls or used LangGraph. Of those 51, 1 in 10 repositories ran premium models in their test suites, 28 instances in total, and the largest single case was 11 instances in one repository with 5,745 stars. That is hard to see rather than careless: a test path reads as free until somebody prices it, which is what the premium models in tests defect page is about.

An evaluation harness is a test path with a judge bolted to it. Both make calls your application code never issues, which puts them in the same category as the calls you never wrote — captured by the provider bill whether or not anything in your attribution scheme was looking for them.

Say what the measurement did not cover

A discussion opened on the AutoGen repository on 2026-08-20, asking for volunteers to run an interoperability evaluation, bounds its own claim before anyone asks:

This is deliberately not a claim that a new language already saves tokens in general use: measured post-decode model API-input saving on unfamiliar dialogue: 0%; total tokens per safely completed real task: not yet known

That is the shape a savings figure should arrive in. It names which quantity was measured, on which input, and which quantity — the one that matters for cost per run — has not been measured yet. A figure without those two boundaries cannot be audited, and auditing it is the whole exercise: a cheaper model is not automatically a cheaper run.

What to record when you run the comparison

1. An identity per arm, set before the first call. Control, candidate and judge are three different spenders. Attach that to the call, not to a spreadsheet afterwards.
2. A ceiling denominated in dollars. Turns, steps and attempts bound a count. The count is not the thing you are protecting.
3. A price for the judge, derived from its own usage fields. Treat a zero as a missing price row until proven otherwise, because that is what it usually is.
4. Two figures published together: the saving per run, and what the comparison cost to produce. The second amortises over future runs, so say over how many.
5. A decision about where experiment spend lands in your unit cost. Shadow traffic mixed into cost per run moves the metric without changing the product, and separating it later requires the identity from point one.

How this works in Capsera

capsera.init() patches the provider clients at process start, so calls made inside an evaluation harness or by a judge are captured whether or not your application issued them — there is no per-call wiring to remember on a path you only run at test time. A decorator on a function you already have, @capsera.agent(name="judge"), says which agent the calls inside it belong to, so control, candidate and judge separate without separate API keys and without threading a tag through the harness. Budgets are scoped per agent, per team or globally and are checked in-process before the provider call, so an experiment can hold its own envelope and exceeding it raises BudgetExceededError instead of appearing on a statement four weeks later. Capsera stores hashes and token counts, never prompt or response content, which is what makes this usable on an evaluation path without creating a second copy of the material being evaluated.

Whether the candidate's answers were better is a different question, and one we leave alone: Langfuse is open source, free, built for evaluation and trace inspection, and is where the judge-cost question quoted above is being asked. If the comparison has to run across many providers behind one interface, LiteLLM is MIT-licensed, free, covers 140+ providers, and is where three of the five items above landed.