Why your prompt caching isn’t working

Anthropic and OpenAI both discount input tokens they have recently seen, by keying a cache on the request’s prefix. Anthropic requires an explicit cache_control marker on the block to cache; OpenAI applies its discount automatically on repeated prefixes. Both match the same way: byte for byte, from the start. One changed character invalidates everything after it.

That makes prompt construction, not prompt content, the thing that decides what you pay. The pattern below is the most common cost defect in public agent code:

from datetime import date

def build_system_prompt(company: str) -> str:
    # The volatile values sit at the START of the prompt, so no two
    # calls ever share a prefix. Nothing fails. Every call pays full price.
    return (
        f"You are a support agent for {company}. "
        f"Today is {date.today()}. "
        "Follow these rules when answering:\n"
        + LONG_STATIC_RULES   # 2,000 tokens that could have been cached
    )

client.messages.create(
    model="claude-sonnet-4-6",
    system=build_system_prompt(company),
    messages=messages,
    max_tokens=1024,
)

The 2,000-token rule block is identical on every call, but it sits behind a company name and today’s date — so as far as the cache is concerned, every request is brand new. This is what prompt prefix stability means: the discount belongs to the part of the prompt that doesn’t move.

Two checks, one from live traffic and one from code:

From responses: providers report cache usage on every response — Anthropic as cache_read_input_tokens and cache_creation_input_tokens. On a repetitive workload, cache reads should dominate input. If the ratio is near zero, nothing is being reused. Capsera records both fields on every event, so the ratio is visible per agent rather than needing a script.

From code: hash the system prompt at each call site and count distinct hashes. A call site that should produce one hash but produces hundreds is rebuilding its prefix per call. This is exactly the static signal the July 2026 repo scan looked for.

The multipliers, from Anthropic’s published pricing as mirrored in our model registry: a cache read costs about 0.1× base input, and a cache write about 1.25×. Two consequences fall out of the arithmetic:

First, caching pays for itself on the firstreuse — one write plus one read (1.25 + 0.1 = 1.35×) is already cheaper than two uncached sends (2.0×). Second, a broken cache costs you roughly 90% of the stable prefix’s input price on every call. For a 2,000-token instruction block on an agent making thousands of calls a day, that is the difference between paying for 2,000 tokens once and paying for them every single time.

We deliberately don’t publish a “typical dollars wasted” figure: it depends entirely on prefix length and call volume, and an invented average would be less useful than the two multipliers above and your own traffic.

In our scan of 133 public Python agent repositories, 76% of the 51 making direct LLM calls assembled system prompts by interpolation — 347 call sites across 39 repositories, the most common defect by an order of magnitude. The bound matters: interpolation is an opportunity signal, not proof of waste, since some of those prompts are deliberately structured with the static part first. The defensible claim is that these prompts defeat caching unless someone has thought about byte order.

Restructure so everything static comes first and everything volatile comes last, then mark the static block cacheable:

from datetime import date

# Static block first, marked cacheable. Volatile facts go AFTER it —
# the cache matches the prefix, so everything before the first changed
# byte is still served at cache-read pricing.
system = [
    {
        "type": "text",
        "text": LONG_STATIC_RULES,            # identical on every call
        "cache_control": {"type": "ephemeral"},
    },
    {
        "type": "text",
        "text": f"Customer: {company}. Today is {date.today()}.",
    },
]

client.messages.create(
    model="claude-sonnet-4-6",
    system=system,
    messages=messages,
    max_tokens=1024,
)

The volatile facts still reach the model — they just stop invalidating the 2,000 tokens ahead of them. If a value doesn’t need to be in the system prompt at all (a date the model could be told in the user turn, a request ID it never uses), moving it out entirely is simpler still.

After deploying, verify from the same signal you diagnosed with: cache reads should jump from near zero to the size of the static block on every repeat call.

Why isn't my Anthropic prompt caching working?

The most common cause is that the prompt's opening bytes change between calls. Provider caches match an exact prefix from the first character, so a system prompt built with an f-string — a date, a user name, a request ID anywhere in it — produces a different prefix every time and never hits the cache. Check the cache_read_input_tokens field on your responses: if it is zero or near zero on a repetitive workload, your prefix is unstable.

How much does prompt caching actually save?

On Anthropic's pricing, a cache read costs about a tenth of base input and a cache write about 1.25 times base input, so a cached prefix pays for its one write the first time it is reused and saves roughly 90% of that prefix's input cost on every call after that. The saving scales with prefix length and call frequency — a long system prompt on a hot path is where it is largest.

How common is broken prompt caching in agent code?

In Capsera's July 2026 scan of 133 public Python agent repositories, 76% of the 51 making direct LLM calls assembled system prompts by interpolation — 347 call sites across 39 repositories. Interpolation defeats shared-prefix caching unless the prompt is deliberately structured so the static part comes first.

• Context regrowth — re-sending the whole history every turn
• Tool result bloat — oversized tool output paid for on every subsequent turn
• All cost defects