LlamaIndex

Query engines, chat engines, and indexing are recorded. Separating retrieval cost from generation cost, and why indexing needs its own agent.

Recorded with no integration. LlamaIndex's OpenAI and Anthropic wrappers construct the provider clients the SDK patches, and OpenAIEmbedding uses the patched embeddings surface.

pip install llama-index capsera

Query and chat engines

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
import capsera

capsera.init(api_key=os.environ["CAPSERA_API_KEY"])


@capsera.agent("docs-indexer", team="knowledge", task_type="indexing")
def build_index(path: str) -> VectorStoreIndex:
    documents = SimpleDirectoryReader(path).load_data()
    return VectorStoreIndex.from_documents(documents)


@capsera.agent("docs-qa", team="knowledge", task_type="retrieval")
def answer(index: VectorStoreIndex, question: str) -> str:
    return str(index.as_query_engine().query(question))

Coverage

Call	Recorded
`index.as_query_engine().query(...)`	Yes
`await engine.aquery(...)`	Yes
`index.as_chat_engine().chat(...)`	Yes
`await engine.achat(...)`	Yes
`VectorStoreIndex.from_documents(...)` embeddings	Yes

Give indexing its own agent

Note that build_index above has a separate agent, which is the recommendation rather than incidental style.

Indexing is usually the largest LLM cost in a RAG system. Embedding a corpus is thousands of calls in a burst, while answering a question is one or two. Combined into one agent, a re-index is indistinguishable from a busy day of queries, and re-indexing on every deploy multiplies the bill invisibly.

Separate agents make the two comparable, and task_type="indexing" against task_type="retrieval" makes them filterable.

Retrieval and generation

A query engine makes embeddings calls to find documents, then a completion to answer. Under one decorator both are attributed to the same agent, which is usually correct.

To split them:

with capsera.tag("retriever", task_type="embedding"):
    nodes = retriever.retrieve(question)

with capsera.tag("synthesizer", task_type="generation"):
    answer = synthesizer.synthesize(question, nodes)

Embeddings bypass routing and budget enforcement, so a blocking budget will not stop an indexing run. To cap indexing, count documents before starting or guard the call yourself.

Global Settings

LlamaIndex configures models globally:

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI

Settings.llm = OpenAI(model="gpt-4o-mini")

This is recorded like any other client. Settings.llm can be assigned before init() runs and calls through it are still recorded, because the patch is on the class the object was built from.

Streaming

streaming=True on a query engine produces a streaming response, and coverage follows the underlying provider. OpenAI and Anthropic streaming is recorded. The synchronous-only providers are not. A stream abandoned before completion has no usage to record. See Troubleshooting.

Which provider is recorded

The wrapper you used. llama_index.llms.openai records openai and llama_index.llms.anthropic records anthropic. An OpenAI wrapper pointed at a compatible vendor records that vendor by base URL rather than the wrapper's name.