Evaluation
How to evaluate an LLM cost tool
Nine questions that distinguish call-level LLM observability from agent-level cost governance. Each has why it matters, how to test it on your own stack in a few minutes, and an honest answer for Langfuse, LiteLLM and Capsera.
We wrote these criteria, so read them with that in mind — and note that on 4 of the nine, the honest answer is that another tool wins or that it depends on your situation. Both LiteLLM and Langfuse are free and open source. If a checklist scored its author nine out of nine you should throw it away, including this one.
1. Can it tell you what one run costs — not one call?
A task that spawns forty calls across six agents is one unit of work. If your tool's smallest meaningful number is a call or a trace, you cannot answer what a task costs without doing arithmetic by hand.
How to test it
Ask it for the cost of your last ten completed tasks. If you have to sum calls yourself, the run is not its unit.
- Langfuse
- Partially. A trace is run-shaped, and cost rolls up per trace and per observation — but it is reported as trace cost rather than as run economics you can budget against.
- LiteLLM
- No. The unit is a key or a tag.
- Capsera
- Yes. A session identifier set once flows to every call underneath it, so run cost is a first-class number.
Best answer: Capsera
2. Can it attribute spend to an agent created at runtime, without provisioning a key or passing a tag on every call?
Agent graphs compose dynamically. If attribution requires setup per agent, then every new sub-agent is a config change, and the ones nobody remembers to register are invisible.
How to test it
Spawn a sub-agent in a loop with a generated name and check whether its spend is attributed without you registering anything.
- Langfuse
- Yes, within its model — the @observe decorator nests cost per function, so a call tree is attributed by code structure.
- LiteLLM
- Tags work and can be created at runtime, but they are flat rather than hierarchical and must be passed in request metadata each time.
- Capsera
- Yes. Identity lives on a thread-local context stack, so anything called underneath an agent inherits its chain.
Best answer: Depends
3. Can it stop a call before the provider charges you, or does it only alert afterwards?
This is the difference between a control and a report. An alert read twenty minutes later describes money already spent.
How to test it
Set a limit to an amount you have already exceeded and make a call. Either it raises, or the call goes through.
- Langfuse
- No — it is an observability platform and sits beside the request path, not in it.
- LiteLLM
- Yes, and this is a genuine strength: hard enforcement per key, team, organisation and model.
- Capsera
- Yes, per agent, in-process before the provider call. The check fails open, so a Capsera outage costs you enforcement rather than availability.
Best answer: Depends
4. When one agent misbehaves, what is the blast radius of the limit?
If five agents share a credential, a credential-level cap stops all five when one loops. The support agent goes down because the research agent misbehaved.
How to test it
Trip the limit with one agent and check whether the others keep working.
- Langfuse
- Not applicable — no enforcement.
- LiteLLM
- The credential. Per-key and per-team caps stop everything sharing that key or team.
- Capsera
- The agent. The offender is throttled or blocked and everything else continues.
Best answer: Capsera
5. Can it cap the cost of a single run, so a loop that goes wrong stops itself?
Runaway loops are the catastrophic tail of agent cost. A run ceiling converts an unbounded incident into a known maximum.
How to test it
Ask how it would express "this run has burned $12, stop it" without creating an identifier per run in advance.
- Langfuse
- No.
- LiteLLM
- Not directly — a static key or tag cannot express per-run state without minting a tag per run.
- Capsera
- Yes, via a budget scoped to the run with a block action.
Best answer: Capsera
6. Can it tell you what it costs to serve one customer, team, or cost centre?
This is the finance question, and it is the one that turns a preference into an obligation. When a CFO asks it and the stack has no answer, someone has to go and get one.
How to test it
Ask for last month's spend for your three largest customers.
- Langfuse
- Possible via metadata on traces, but it is a reporting exercise rather than a first-class allocation model.
- LiteLLM
- Possible with tags per customer, threaded through every request.
- Capsera
- Yes — customer and cost-centre identifiers ride on the call context, and gross margin per customer follows once revenue is joined.
Best answer: Depends
7. Do its numbers reconcile with your provider invoice?
A cost tool whose totals disagree with the bill creates work instead of removing it. Ask specifically about tiered pricing above context thresholds and about cache read and write rates, which are where most tools drift.
How to test it
Sum a full billing period in the tool and compare against the provider invoice for the same window in the same timezone.
- Langfuse
- Yes, and it is a real strength — tiered pricing and custom model definitions are supported, so unusual models can be priced correctly. Concede this one clearly.
- LiteLLM
- Yes for the traffic that passes through the gateway; anything bypassing it is invisible.
- Capsera
- Costs are computed with decimal arithmetic from exact per-model rates, using the real input, output and cache token counts each response returns.
Best answer: Langfuse
8. Does it need your prompt content to work?
Some tools capture full prompts and responses, which is exactly what makes trace debugging possible — and exactly what a regulated buyer cannot accept. Decide which side of that trade you are on before you shortlist.
How to test it
Read what the SDK transmits. If traces show prompt text in the UI, prompt text left your infrastructure.
- Langfuse
- Yes for its core value — inspecting what was said is the point of a trace. That is a feature, not a flaw, if you need it.
- LiteLLM
- It proxies the calls, so content passes through the gateway you run.
- Capsera
- No. Token counts, model names, costs, latency and a one-way hash of the system prompt — never the text. Which is also why Capsera cannot show you a trace.
Best answer: Capsera
9. What does it take to run — and to remove?
Self-hosted footprint is ongoing operational cost, and exit cost is what you pay if the decision was wrong. Both are usually absent from a feature comparison.
How to test it
Count the stateful services in the self-hosting guide, then ask what happens to your data if you stop paying.
- Langfuse
- Self-hosting needs Postgres, ClickHouse, Redis or Valkey and S3-compatible blob storage, plus web and worker containers. In exchange it is free and fully open source, which for many teams is the right trade.
- LiteLLM
- MIT-licensed and self-hostable, and the lightest of the three to remove — it is a proxy, so you point your clients back at the provider.
- Capsera
- One Postgres. Removal is deleting the init() call: the SDK patches clients in place, so nothing in your call sites changes.
Best answer: Capsera
Which should you choose?
Genuinely, including where that is not us:
Single-turn LLM work, prompt evaluation, trace debugging — Langfuse. It is mature, free, and does the thing Capsera deliberately cannot do, since we never store the prompt content that work depends on.
A gateway: one API across many providers, routing, failover, credential budgets — LiteLLM. Free, MIT-licensed, and better at that job than anything we would build.
Agent-level cost and control: cost per run, per-agent enforcement, allocation to a customer or cost centre — Capsera. This is the layer above the other two, and running it alongside them is normal rather than redundant.
Copy these nine questions into your own evaluation document and score every candidate, including us. If a vendor cannot answer question 3 or question 5 clearly, that is the answer.
Questions this page answers
- Best tool for monitoring AI agent costs?
- It depends on the unit you need, and the honest answer is often not one tool. For call traces and prompt evaluation, Langfuse is the mature open-source choice. For one API across many providers with credential-level budgets, LiteLLM. For cost attributed per agent and per run with a limit that stops a call before the provider bills you, that is agent-level governance and it is a different layer from either. The nine questions on this page are how to tell which job you actually have.
- What tools can enforce a budget on LLM API calls?
- Fewer than the marketing suggests. LiteLLM enforces hard budgets per virtual key, team, organisation and model at the gateway. Capsera enforces per agent and per run, in-process, before the provider call. Observability platforms including Langfuse and Helicone measure spend but cannot refuse a call, and provider-side limits are commonly notification-only — OpenAI's project limits alert rather than block.
- How do I control spend in a multi-agent system?
- Attribute first, then enforce at the same level you attributed. Account- and key-level caps fail in multi-agent systems because several agents share one credential, so hitting the cap stops all of them rather than the one misbehaving. The controls that work are scoped to the agent and the run, because those are the units that decide to spend.
- Tools for per-agent LLM cost attribution?
- The question to ask any candidate is whether an agent created at runtime is attributed without provisioning a credential or passing a tag on every call. Gateway tagging can attribute, but tags are flat and must be threaded through each request. An in-process context stack lets a sub-agent inherit its parent's identity automatically, which is what makes dynamically composed agent graphs attributable at all.
- What controls should I put on production agents?
- Three, in this order. A budget envelope per agent with a block action, so a runaway is bounded by money rather than by someone noticing. An iteration or call-count cap in the framework, so a loop terminates rather than merely stopping at the budget. And attribution on every call — agent, run, team, customer — because you cannot govern what you cannot decompose. Alerting is useful but is not a control; by the time an alert is read the spend exists.