Prompt analysis

Optional structural metrics that identify cost problems in prompts, such as uncached preambles, tool-result bloat, and context regrowth, without recording content.

Token counts show that a call was expensive. Prompt analysis records the shape of the request, which shows why.

It is off by default, because it does work on every call.

capsera.init(
    api_key=os.environ["CAPSERA_API_KEY"],
    enable_prompt_analysis=True,
)

Recorded metrics

Metric	Identifies
system prompt hash	the same preamble re-sent across many calls
estimated prompt tokens	prompt size independent of the response
message count, conversation turns	history growing without bound
few-shot detection and pair count	examples inflating every call
tool-result tokens	tool output dominating the prompt
context-window utilisation	how close to the model's limit you are running

All of these are structural. No prompt text, messages, tool arguments, or tool results are recorded. See Privacy.

The system-prompt hash lets the backend detect that many calls shared one preamble, which is how an uncached repeated block becomes visible. It is one-way and cannot reproduce the prompt.

Problems it identifies

An uncached repeated preamble. If one system prompt hash appears across thousands of calls with no cache reads, you are paying full input price for the same tokens every time. The fix is prompt caching.

Tool-result bloat. An agent that appends full tool output to its history grows its prompt every turn. High tool_result_tokens against a modest response size indicates this. Truncate or summarise tool output.

Context regrowth. Conversation turns climbing steadily means history that is never trimmed, so cost per call rises with turn count.

Unnecessary few-shot examples. Examples added during prototyping are paid for on every production call. Detection requires 70% of assistant replies to be short, so multi-turn conversations are not misread as few-shot examples.

Running near the context limit. High utilisation is a reliability problem as well as a cost one, because the next long input truncates or fails.

Oversized-prompt callback

def on_big_prompt(tokens: int, threshold: int) -> None:
    logger.warning("prompt of %d tokens exceeded %d", tokens, threshold)

capsera.init(
    api_key=...,
    enable_prompt_analysis=True,
    prompt_size_threshold=8_000,      # default 4_000
    on_oversized_prompt=on_big_prompt,
)

This works as a regression guard in CI, failing the build if a change pushes a prompt past an agreed size.

Estimates and real counts

Prompt-analysis token figures are estimates computed locally without a tokenizer. They are stored in their own fields and never replace the provider's counts, which is what cost uses.

Use them to compare prompts against each other and to detect growth, not to reconcile a bill.

Runtime cost

Analysis walks your messages and estimates sizes on every call, before the provider call, which is why it is off by default.

A practical approach is to enable it temporarily, for a day or in one environment, identify the problems, fix them, and disable it again. The problems it finds are structural rather than fluctuating.

Not applied to embeddings

Embedding calls bypass prompt analysis, along with routing and budget enforcement.