Where the model runs changes what the loop costs

An agent loop re-sends its entire transcript on every turn, so the bulk of what a long run spends is input it has already paid for at least once. Whether that re-send is discounted as a cache read or charged as fresh input is decided by the stack serving the model — the provider's own cache, a gateway in front of it, or the runtime on a box you rent — and not by anything in your code. The same agent, on the same weights, sending the same bytes, has a different cost curve on either side of that move. And the move is almost always made for latency, privacy or jurisdiction, by someone who was not asked about cost.

The cache belongs to the serving stack, not to your prompt

A commenter benchmarking a new on-device model through a local runtime on 2026-10-15 named the failure in one line:

The problem is ollama is not caching the prompt, I get -> forcing full prompt re-processing due to lack of cache data

They drew the conclusion themselves in the next sentence: "So this wouldn't work in its current state in an agentic loop." The same comment reports the hardware's throughput — "~45 tokens/s generation and ~680 tokens/s prompt processing" — which is the number that matters here, because with no cache every turn pays the prompt-processing rate over the whole transcript rather than over the new part of it.

Nothing in that setup is misconfigured and no prompt is at fault. Prompt prefix stability is the half of prompt caching your code owns: keep the opening bytes identical across calls and a cache is able to match them. The other half is whether there is a cache on the other end that will. That half is a property of the serving path, it is not visible in your repository, and it can change without a deploy.

The path moves for reasons that are not about cost

A launch on 2026-10-13 for a gateway aimed at regulated industries states its purpose plainly: "We route all AI requests to ITAR restricted and controlled unclassified information (CUI) compliant infrastructure." That is a compliance decision, and it relocates every request in the system.

A commenter in a thread about a newly released open-weights model on 2026-10-18 gave the other version of the same move: "We can just get a box of compute even in Germany, and run the exact same model here without anybody stopping us." Also correct, also about jurisdiction, and also a relocation of every request.

Both are reviewed as architecture changes. Neither is reviewed as a change to what an agent loop costs per turn, because on the face of it nothing about the agent changed. The model id can even stay the same. What changed is who holds the prefix between turn four and turn five.

An uncached loop pays for its history at full rate, every turn

Context regrowth is the shape of the problem already: a stateless API means turn twenty re-sends the previous nineteen, so input tokens grow roughly with the square of the conversation length. Prompt caching is the thing that blunts it — the re-sent part is matched rather than re-read, which is why a loop on a cached path has a cost per turn that flattens after the first one.

Take the throughput above as arithmetic rather than as a benchmark. At about 680 tokens per second of prompt processing, a 40k-token transcript takes roughly a minute of pure re-reading before the first output token of that turn exists. Ten turns at that size is ten minutes the machine spends reading what it already read, and a coding loop reaches that size early. On a cached path most of those tokens would have been matched instead.

When you own the box, that time is the bill. There is no invoice line and no per-token rate, so the spend shows up as occupied GPU seconds, as queueing, and as the run that did not start. This is the same substitution covered in what a run costs depends on how the capacity was bought: the basis changes, the quantity does not stop existing. A token count is comparable across both worlds; a dollar figure is not.

Four things to check when the serving path changes

1. Input tokens per turn, split by class. Cache reads, cache writes and uncached input need to stay separate dimensions. A single collapsed "input" number is identical in both worlds, which is precisely why the move is invisible on most dashboards.
2. The curve across turn index within one run, not the average across calls. An uncached loop slopes upward through the session; a cached one steps once and flattens. Averaging the two produces a number that describes neither.
3. How long the new path holds a prefix. A cache with a short lifetime is no cache for a loop that pauses on a slow tool call, and that interval is not something your agent code decides.
4. Whether a prompt change and a path change can be told apart afterwards. If the only record is a daily cost total, a cache that quietly stopped matching and a prompt that grew look the same on the chart.

The defect itself is not new — it is the one written up at broken prompt caching. What is worth separating is the cause: a prefix your own code destroyed, versus a prefix nothing on the other end was ever going to match.

Where Capsera sits, and what it does not see

Capsera runs in-process, which is the side of this line where the turn is still attributable. capsera.init() patches the provider clients — Anthropic messages, OpenAI chat completions, OpenAI embeddings and Google Gemini, sync and async — so each call is recorded with the agent that made it, and prompt analysis reports cache hit rate and context-window utilisation as structural metadata, without storing prompt or response content. That is what makes the per-turn curve in point 2 readable per agent rather than per account.

Three limits, stated. The patched surfaces are the ones named above: a local runtime driven through its own client library is not one of them, so confirm what your serving path's client actually is before assuming the numbers are there. Budgets are scoped per agent, per team or globally and are checked before the provider call; a per-run cost ceiling is a category concept we write about and not a shipped feature. And a budget denominated in dollars has nothing to price against on hardware you already paid for — there, the quantity worth bounding is tokens per turn, not spend.

Two places to look elsewhere. If the reason the path moved is that you need to run many providers behind one interface, LiteLLM is MIT-licensed, free, covers 140+ providers and enforces hard budgets per key, team, organisation and model — that is the layer the relocation belongs in, and it is better at it than we are. If what you need is to compare the content of turns before and after the move, Langfuse is open source, free and built for trace inspection; Capsera stores hashes and token counts and never the text. Which layer answers which question is set out in gateway vs observability vs governance.