A memory layer trades prompt size for turns

A memory or retrieval layer is installed to keep an agent's context small, and the saving it produces is measured exactly where it is applied: on the call whose prompt got shorter. What it costs is paid across the rest of the run — in the store's own billed calls, in the turns the model spends checking what came back, and in a retrieved block that changes on every turn where a stable prefix used to be. The quantity that settles it is cumulative input tokens per completed run, with the layer on and off. Tokens on the call you shrank will report success whether or not the run got cheaper.

This is the neighbouring case to what compaction costs. That post is about summarising a transcript inside one run. This one is about the layer you install so the transcript never grows that far.

What one developer measured when they checked

An Ask HN thread on 2026-11-10, on managing agent context across clients and projects, states the motive and the result in the same comment. The motive first:

I've been trying different approaches to tackle agent memory across different sessions, without bloating context and increasing token spending.

Then the measurement, from the same comment:

when I decided to measure token usage, I ran into a surprising +23% in token usage (model was often double checking or query was finding more matches than intended)

That is one developer's figure, on their own benchmark and their own setup. The same comment puts a second approach — a static graph of codebase structure — at +5% on the same comparison. Neither number transfers to your agent. What transfers is the pair of causes named in that parenthesis, because both are structural: the model double-checked what it was handed, and the query returned more than was wanted.

The layer itself is no longer unusual. A submission the same week, arguing that agent memory needs its OpenTelemetry moment, treats it as infrastructure to be standardised rather than a technique, and the first reply asks which observability platform to point at it. The pain it is installed against is described plainly by a commenter in a model-release thread on 2026-11-10 — that cruft accumulates and "creates back pressure via growing token usage", and that the model "doesn't learn, though, so you pay for those indirections every time".

The saving is on one call, and the cost is spread over the run

The store bills on both sides. Writing a memory costs a call, usually an embedding and often a summarisation. Reading one costs a query, and some designs spend a model call deciding what to retrieve before retrieving it. None of those appear at your own call sites, which puts them in the same category as the calls you never wrote: billed, in-process, and absent from any attribution scheme built by wrapping the code you typed.

A retrieved fact invites a check, and a check is a whole turn. Every turn re-sends everything the run has accumulated, so a verification turn triggered by a two-hundred-token memory block late in a run is billed at the size of the entire transcript, not at the size of the block. That asymmetry is the heart of it: the saving is bounded by the context you removed, and the cost is a multiple of the context you kept.

Retrieval returns what matches, not what was wanted. An over-matching query hands back a block larger than the material it replaced, and nothing in the stack reports that as a failure. The run completes, the answer looks fine, and the prompt is bigger than it was before the layer was added.

The retrieved block is the part of the prompt that moves

Provider prompt caching matches a prefix from the first byte — see prompt prefix stability for what that depends on. A memory block interpolated into a system prompt makes that system prompt dynamic: content that changes every turn, placed ahead of content that never changes, so every byte after it stops matching what was cached.

That pattern is common and it is hard to see. We scanned 133 public Python agent repositories; 51 made direct LLM calls or used LangGraph, and across 39 of them we found 347 call sites that build system prompts in ways that defeat shared-prefix caching unless deliberately structured (method and limitations, and the defect itself at broken prompt caching). A memory layer adds one more of them, in the place where it does the most damage.

The fix is placement rather than removal. Stable instructions and tool definitions first, retrieved material last, so a block that changes each turn invalidates only what follows it instead of the whole prompt.

The comparison has to be a before and after on the same tasks

A memory layer arrives mid-project, so the comparison most teams actually have is this month against last month — different tasks, a different prompt, often a different harness version. That comparison cannot answer the question.

Per-call averages are worse than useless here, because the layer adds short calls and short turns to the denominator. A retrieval query is a cheap call. Add enough of them and the average cost per call falls while the total for the run rises, which is the exact shape a cost review reads as a win.

What to record

1. Cumulative input tokens per completed run, with the layer on and off, over a fixed set of tasks. This is the number the decision rests on.
2. Billed calls per run, split by who issued them — work calls against memory and retrieval calls. A single total cannot separate the layer from what it supports.
3. Turns per completed run, as a distribution rather than a mean. Verification turns show up here before they show up anywhere financial.
4. The size of the retrieved block per turn, at its maximum. Over-matching is a tail event, and an average hides exactly the runs that cost the most.
5. Cache reads, cache writes and uncached input as separate dimensions. A block that moves converts reads into writes, and one combined input figure deletes the evidence of it.

How this works in Capsera, and what it does not do

capsera.init() patches the provider clients — Anthropic messages, OpenAI chat completions, OpenAI embeddings and Google Gemini, sync and async — so the embedding and summarisation calls a memory layer makes inside your process are captured without being wired up one at a time. Putting @capsera.agent(name="memory") on the function that does the retrieval gives the layer its own identity, and because the agent stack is a contextvars ContextVar rather than thread-local state, nested calls inherit it. The layer's spend then separates from the work it was installed to support, which is the split the whole question turns on.

The limitation is worth stating plainly. Capsera stores hashes and token counts, never prompt or completion content, so it cannot tell you what was retrieved or whether it was relevant. It can tell you that the block grew, that the run took four more turns than the same task took last week, and which agent paid for both. Budgets are checked before the provider call and scoped per agent, per team or globally; there is no per-run ceiling, and enforcement fails open, so a run that drifts upward is bounded by the agent's envelope rather than by its own.

If what you need is the content of the retrieval step — what came back, not how big it was — Langfuse is open source, free, and built to store and inspect exactly that.