Cost per run depends on what counts as one run

Cost per run is a ratio, and almost all of the effort goes into the numerator — token counts, price tables, rollups. The denominator is where the same spend turns into different numbers: whether a retried call is one operation or two, whether a run stopped at a step cap counts as finished, and whether a cancelled job counts at all. Three changes in the past week, across two projects, each had to answer one of those questions in code before anything could be reported.

A retry is one operation and two attempts, and both were billed

A Langfuse discussion from 2026-08-21 puts the case in three sentences (usage semantics: should traces count operations or attempts?):

An agent retries the same provider call once, then succeeds. Two telemetry records exist. What should usage metrics say?

The options it lays out are one operation with two attempts, or two operations. This is not a philosophical question about traces; it decides what a per-call average means. Whether the failed attempt was billed depends on how far it got before it failed, which is exactly why the two counts have to be recorded separately rather than derived from each other. A pipeline that stores only "calls" cannot later tell you which kind it counted.

The write side already knows. A LiteLLM pull request from 2026-08-20 that gates shadow-eval budgets on dollars instead of turns describes its row unit plainly:

Each attempt records the shadow arm's and judge's billed cost on its row

The attempt is the natural unit at write time, because an attempt is what a request is. An operation is a rollup over attempts — and a rollup is only possible if every attempt row carries the identifier of the operation it belongs to. Where that identifier is missing, the two numbers are not recoverable from each other in either direction, which is the same schema argument as a daily cost rollup cannot answer per-run questions, one level further down.

A run that was cut off can still be recorded as a success

A Langfuse fix from 2026-08-19 surfaces step-limit truncation instead of silent success:

When the assistant hits the 20-step cap mid-loop, the run no longer looks like a clean success. Postgres records SUCCEEDED with error_code = step_limit, the activity header reads Stopped after …, and the chat shows a warning to send another message.

Read that from the cost side. A step cap is a control on runaway agent loops, and it works: the spend stops. But if a truncated run lands in the completed bucket, the cap also improves cost per completed task, and it improves it precisely when the agent is being cut off before finishing the work.

The metric then moves the wrong way in both directions. Raise the cap and runs cost more and look worse. Lower it and runs cost less and look better, while fewer tasks are actually done. A number that responds like that to a configuration change is not measuring the thing its name claims. Cost per completed task is comparable week to week only when "completed" means the task finished, not that the harness stopped calling.

Succeeded and failed are not the only endings

A LiteLLM pull request from 2026-08-19 adding enqueued-token rate limiting for batches reserves an estimate up front and releases it afterwards:

Reservation is refunded when the batch completes, fails, expires, or is cancelled

Four terminal states for one submission. A boolean success column holds two of them and silently maps the other two onto whichever value the code defaults to. Expired and cancelled runs are the awkward pair: they spent real money and produced nothing to divide it by, so they belong in the numerator and in a separate line of the denominator, never in the completed count. The reservation pattern itself is worth noting as the general answer to the problem in a budget check cannot price the call it allows — hold the worst case, then release what was not used.

The boundary of a run is not the boundary of a process

A comment on Hacker News on 2026-08-19, under a launch thread for a persistent-VM product, asks where the boundary actually is (thread):

One question on the agent fleet pattern you described: when a pilot agent delegates to dozens of sub-agents across separate VMs, how do you track what the whole job actually cost? The VM minutes are visible, but the API calls each agent makes to OpenAI, Anthropic, Serper, Firecrawl, those are spread across processes and vendors.

A run is defined by the work a user asked for, not by a process boundary, a container or a vendor. If the identifier is minted where the job starts and carried into everything the job spawns, the run is a sum. If each process invents its own, the same work arrives as a dozen unrelated runs and the average cost per run is an artefact of how the work was scheduled. Calls your own code never issued sit inside the same boundary and have to be counted there too — agent cost includes the calls you never wrote.

What to write down when a run ends

1. One identifier, minted at the top of the job, carried into every process, sub-agent and framework call underneath it.
2. Two counters, not one. Attempts and operations are different numbers. Store the attempt row with the operation identifier on it, and derive whichever you need at read time.
3. A terminal state with more than two values: completed, failed, cancelled, expired, stopped at a limit. The reason it stopped is part of the record, not a detail for the logs.
4. Accumulate cost over attempted runs; divide by completed ones. Report both figures and the ratio between them. The gap is a reliability signal, and hiding it flatters the unit cost.
5. Never let a cap count as a completion. A step, turn or budget limit changes what stopped the run, not what finished it, and it should be visible in the run's record as its own ending.

How this works in Capsera

Capsera groups calls into a run with a session identifier set once, and attributes each call to the agent that made it through a decorator on a function you already have, so a run total is a sum over its own calls rather than a slice of a daily figure. What Capsera cannot know is whether your run achieved anything: the terminal state — completed, cancelled, stopped at a limit — is a fact only your code has, so write it against the session identifier and join on that. Budgets are checked in-process before the provider call and are scoped per agent, per team or globally. Capsera stores hashes and token counts, never prompt or response content.

If the question you are actually asking is about the content of a trace — which step retried, what the model was sent, whether the output was any good — Langfuse is open source, free, built for it, and is where the discussion quoted at the top of this post is happening. For gateway routing and provider-level budgets across many providers, LiteLLM is MIT-licensed, free, and covers 140+ providers.