How is agent observability different from LLM observability?

The unit of analysis. LLM observability is organised around the request: it tells you what one call sent, what came back, how long it took and what it cost. Agent observability is organised around the run — one task that may have spawned forty calls across six agents, where no individual call explains the outcome or the bill. Both are useful, but a per-request view cannot answer "what did this task cost" without a run identifier tying its calls together.

What's the difference between an LLM gateway and LLM observability?

A gateway sits in the request path: it proxies calls to providers, gives you one API across many of them, and can enforce limits on the credentials passing through. Observability sits beside the request path: it records what happened for later analysis and cannot refuse a call. The practical distinction is that a gateway can say no and an observability tool can only report, so they are usually complementary rather than alternatives.

Which LLM observability tool should I use?

It depends which job you actually have. For prompt evaluation, trace debugging and dataset experiments, Langfuse is the mature open-source choice and handles cost computation accurately including tiered pricing. For one API across many providers plus credential-level budgets, LiteLLM is the standard. If what you need is cost attributed per agent and per run with a limit that stops a call before the provider bills you, that is a governance job and neither of those layers does it.

Compare

Gateway, observability, governance

These are three different layers, not three competitors. A gateway sits in the request path and decides which provider serves a call and whether the credential still has budget. Observability sits beside the request path and records what each call did. Governance runs in your process before the call and decides whether a specific agent, on a specific run, is allowed to spend. Most teams past a certain size end up with more than one, and the layers stack rather than replace each other.

The three layers

Layer	Where it sits	Unit	Can it refuse a call?
Gateway	In the request path	The credential — key, team, organisation, or tag	Yes, at the credential level
Observability	Beside the request path	The request, grouped into traces	No — it records after the fact
Governance	In-process, before the call	The agent and the run	Yes, per agent and per run

Named examples, so this is checkable rather than abstract: Gateway — LiteLLM, Portkey · Observability — Langfuse, Helicone, LangSmith · Governance — Capsera.

What a gateway does

A gateway terminates your provider calls and re-issues them. That position is what lets it do three useful things: present one API across many providers, route or fall back between them, and enforce a budget on the credential a call arrives with.

LiteLLM is the reference implementation and it is MIT-licensed, supports well over a hundred providers, and enforces hard budgets per virtual key, team, organisation and model, with tag budgets creatable at runtime. That enforcement is real: a key over its limit gets refused.

The limit is the shape of the unit rather than the strength of the block. Tags are flat rather than hierarchical and have to be passed on each request, so expressing “this agent, inside this run, inside this team” means minting and threading identifiers yourself. And a credential-level cap has a blast radius: if five agents share a key, the cap stops all five when one of them misbehaves.

What observability does

Observability records. It captures what a call sent and returned, groups calls into traces, computes cost from usage, and gives you somewhere to look when output is wrong or spend is surprising.

Langfuse is the mature open-source option and is genuinely good at the cost part — it handles tiered pricing and custom model definitions, and its @observe decorator nests cost per function, so you get a hierarchy rather than a flat list. It is also free, which matters when you are comparing.

What it structurally cannot do is refuse a call, because it is not in the path. It also needs your prompt and response content to do its main job — trace debugging is inspecting what was said — which is a reasonable trade for evaluation work and a problem if your constraint is that prompt content must not leave your infrastructure.

What governance adds

Governance answers a question neither other layer is shaped for: what did this run cost, which agent caused it, and should the next call happen at all.

The unit is the reason it is a separate layer. A gateway sees credentials; an observability tool sees requests. Neither sees the run — the task that spawned forty calls across six agents — unless something ties those calls together as they are made. Capsera does that in-process: an agent’s identity lives on a context stack, so a sub-agent created at runtime inherits it with nothing passed per call.

Being in-process is also what makes pre-call enforcement possible without a proxy: the check runs before the provider call, and a budget with a block action raises inside your application instead of sending the request. The trade is stated plainly — the check fails open, so if Capsera is unreachable the call proceeds and you lose enforcement for that moment rather than availability.

Which do you actually need?

Honestly, and including the cases where the answer is not us:

Evaluating prompts, debugging why a model said something, running dataset experiments — use Langfuse. This is its core competence, Capsera does not do it, and we never store the prompt content that work depends on.

One API across many providers, routing, failover, credential budgets — use LiteLLM. It is free, mature, and better at the gateway job than anything we would build.

Cost per agent and per run, a limit that stops a runaway before the provider bills you, spend attributed to a customer or cost centre — that is the governance layer, and it is what Capsera is for.

These combine. A common shape is LiteLLM as the gateway, Langfuse for evaluation, and governance above both — because the questions each answers do not overlap as much as the category names suggest.

Questions this page answers

How is agent observability different from LLM observability?: The unit of analysis. LLM observability is organised around the request: it tells you what one call sent, what came back, how long it took and what it cost. Agent observability is organised around the run — one task that may have spawned forty calls across six agents, where no individual call explains the outcome or the bill. Both are useful, but a per-request view cannot answer "what did this task cost" without a run identifier tying its calls together.
What's the difference between an LLM gateway and LLM observability?: A gateway sits in the request path: it proxies calls to providers, gives you one API across many of them, and can enforce limits on the credentials passing through. Observability sits beside the request path: it records what happened for later analysis and cannot refuse a call. The practical distinction is that a gateway can say no and an observability tool can only report, so they are usually complementary rather than alternatives.
Which LLM observability tool should I use?: It depends which job you actually have. For prompt evaluation, trace debugging and dataset experiments, Langfuse is the mature open-source choice and handles cost computation accurately including tiered pricing. For one API across many providers plus credential-level budgets, LiteLLM is the standard. If what you need is cost attributed per agent and per run with a limit that stops a call before the provider bills you, that is a governance job and neither of those layers does it.