LiteLLM and gateways

Gateways sit below the SDK rather than beside it. There are two shapes and they behave differently.

LiteLLM as a library

litellm.completion() dispatches OpenAI-family models through the real openai client, so the calls are recorded. Verified in the SDK's test suite against real LiteLLM.

import litellm
import capsera

capsera.init(api_key=os.environ["CAPSERA_API_KEY"])


@capsera.agent("router", team="platform")
def ask(prompt: str) -> str:
    response = litellm.completion(
        model="openai/gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content

LiteLLM is on the frame-skip list, so caller_file points at your function rather than litellm/llms/openai/openai.py. This needed handling specifically, because LiteLLM's dispatch module lives at site-packages/litellm/llms/openai/openai.py, which does not match the site-packages/openai skip fragment.

LiteLLM supports many providers through its own HTTP handlers rather than a vendor SDK. Where it does, there is no patched client in the path and the call is not recorded. OpenAI-family models are the verified path. For anything else, use record().

LiteLLM Proxy and other OpenAI-compatible gateways

A gateway running as a separate service is simpler. Point the openai client at it:

client = openai.OpenAI(
    base_url="https://gateway.internal.example.com/v1",
    api_key=os.environ["GATEWAY_KEY"],
)

Coverage is full, because this is the OpenAI client.

Two things affect what is recorded.

Provider labelling. The SDK identifies providers by base URL, and a private gateway host is not in its map, so the label falls back to the model's catalogued provider. A call through your gateway using gpt-4o-mini records openai, which is correct about the model and says nothing about the gateway. Report the host and it will be added to the map.

Model aliases. Gateways often expose their own names, such as prod-fast or cheap-tier, that resolve to a real model upstream. An alias cannot be priced.

The SDK prefers the model reported in the response over the one requested. If your gateway rewrites the response's model field to the upstream model, cost is correct automatically. If it echoes the alias, you will see a fallback-estimate warning:

capsera: no pricing found for model 'prod-fast' — using fallback estimate

Configuring the gateway to report the resolved model is the single change that makes gateway cost accurate.

Why not integrate at the gateway instead

A gateway sees requests from one API key. It cannot know that a call came from your planner node rather than your summarizer, because nothing in the HTTP request carries that. The information exists only in the calling process's stack, which is why the SDK runs in-process.

The two compose. The gateway handles failover, key management, and caching. Capsera handles attribution and enforcement.