Prompt Cache Misses: Why Your LLM Caching Savings Disappear

You turned on prompt caching and expected the input bill to drop. It didn’t. Or it dropped for a week and crept back up.

Usually the cache isn’t broken. It’s missing. Every miss means you pay full price for tokens you expected to be cheap, and on some providers you also pay a premium to write the cache. This guide covers why misses happen, how to measure your hit rate, and how to calculate whether caching is paying off.

How Prompt Caching Saves Money (and How It Can Cost More)

Caching lets a provider reuse the processed beginning of a prompt instead of recomputing it. Anthropic’s prompt caching docs describe the pricing structure: cache writes cost 1.25x the base input price for a 5-minute lifetime and 2x for a 1-hour lifetime, while cache reads cost 0.1x of base price on most models. A few newer models have even cheaper reads, so check the per-model table.

That structure has a catch. A write costs more than a normal request, and the discount only appears on later reads. If reads never come, you paid the write premium for nothing.

OpenAI works differently. OpenAI’s guide says caching is automatic for prompts of 1,024 tokens or more, with no code change needed, and cached tokens show up in a cached_tokens usage field. The same guide notes that newer model families report cache writes separately and bill them at a cache-write rate, so check the rate for your model.

The Break-Even Calculation

Let w be the write multiplier and r the read multiplier. Reusing the same prefix across k requests within the cache lifetime pays off when:

k > (w − r) ÷ (1 − r)

With a 5-minute write (1.25x) and a 0.1x read, that gives k > 1.28. In plain terms, caching pays off from the second request that hits. With a 1-hour write (2x), it gives k > 2.11, so you need three uses.

Here is a worked example using the Sonnet 5.5 prices from Anthropic’s table ($2 per million base input tokens, $2.50 for a 5-minute write, $0.10 for a read). Take a 50,000-token prefix used 10 times:

Scenario

Cost

No caching

10 × $0.10 = $1.00

Caching, 1 write + 9 hits

$0.125 + 9 × $0.005 = $0.17

Caching, every request misses

10 × $0.125 = $1.25

A healthy cache saves about 83% in this example. A cache that never hits costs 25% more than no cache at all. Prices change, so use your own model’s rates, but the shape holds everywhere.

How to Measure Your Cache Hit Rate

Don’t guess. Read the usage fields in each response.

On Anthropic, the response reports cache_read_input_tokens, cache_creation_input_tokens and input_tokens. The docs explain that total input is the sum of all three, and input_tokens only counts tokens after the last cache breakpoint. So:

Hit rate = cache_read_input_tokens ÷ (cache_read + cache_creation + input_tokens)

On OpenAI, compare cached_tokens to total prompt tokens. If both Anthropic cache fields are 0, the prompt wasn’t cached at all, which usually means it was below the minimum length.

Track this number over time, per application or per developer. A hit rate that falls after a deploy points straight at the change that broke it.

9 Reasons Your Prompt Cache Misses

1. The breakpoint is on content that changes

Anthropic’s docs call this the common mistake. If your cache marker sits on a block that includes a timestamp or the user’s message, the prefix hash differs on every request. You pay for a fresh write each time and never get a read. The cache doesn’t look behind your breakpoint for stable content. It only finds entries earlier requests wrote at their own breakpoints. The fix is to put the marker on the last block that stays identical.

2. Something in the prefix changed

Caching needs an exact match. According to the docs, the prefix order is tools, then system, then messages, and a change at any level invalidates that level and everything after it. Modifying tool definitions invalidates the whole cache. Toggling web search or citations, changing the speed setting, changing tool_choice, adding or removing images, and changing thinking or effort settings all break parts of the cache. For coding agents, this means adding or removing tools mid-session can quietly reset your savings. See also what reasoning effort does to cost.

3. The cache expired

The default Anthropic lifetime is 5 minutes. Each hit refreshes it at no extra charge, but the lifetime is measured from the start of the request, so a response that takes four minutes to stream leaves only about one minute for the next request. If your traffic has gaps longer than five minutes, the 1-hour TTL may be cheaper overall, per the docs. OpenAI retention is model-dependent, and its guide describes extended retention of up to 24 hours as an option, so check the current policy for your model.

4. The prompt is too short

Caching is silently skipped below a minimum length, and no error is returned. Anthropic’s minimums range from 512 to 4,096 tokens depending on the model. OpenAI’s is 1,024 tokens. A “cached” system prompt that is too short simply never caches.

5. A growing conversation outran the lookback window

Anthropic checks at most 20 positions back for a matching entry. If a long agent turn adds many blocks between requests, the prior entry falls outside the window and you miss. The docs recommend a second breakpoint closer to the end so a write accumulates there before you need it.

6. Parallel requests started too early

A cache entry only becomes available after the first response starts. If you fire many identical requests at once, they all miss. Send the first, wait for it to begin responding, then send the rest.

7. Dynamic content sits at the front

A request ID or timestamp at the top of the prompt changes everything after it. Put static content first and variable content last. This applies to OpenAI too, where exact prefix matching is required.

8. Different workspace or model

Anthropic caches are isolated by workspace on the Claude API, so splitting traffic across workspaces splits your cache. One OpenAI guide also reports that caches are isolated per model, so switching models mid-session means starting over. Treat that as a third-party claim and test it.

9. Unstable JSON ordering

Anthropic’s docs warn that some languages, such as Swift and Go, randomize JSON key order, which changes tool_use content and breaks the cache. Serialize with a fixed key order.

How to Fix Prompt Cache Misses

1. Use Anthropic’s cache diagnostics. The docs describe a feature that compares consecutive requests and reports where the prefix diverged. Start there.
2. Run the checklist. Confirm the prefix is identical across calls, tool_choice, images, thinking and effort settings haven’t changed, and the prompt clears the minimum length.
3. Move the breakpoint. Put it at the end of the static prefix.
4. Pre-warm the cache. Anthropic supports a request with max_tokens set to 0 that writes the cache without generating output, so your first real user doesn’t pay the miss.
5. Match TTL to your traffic gaps. Use the 5-minute cache when requests arrive more often than every five minutes, and the 1-hour cache for gaps between five minutes and an hour.
6. Track hit rate per team and per model, so a regression shows up the week it happens.

If caching alone can’t get you far enough, compare it with other levers in prompt caching vs prompt compression. Hidden thinking tokens aren’t discounted by caching either, as covered in what reasoning tokens add to your bill.

Make Cache Savings Visible Across Your Team

Cache behavior varies by developer, tool and model, and a single bad setting can erase savings across a team. To see it, you need cost and token breakdowns by user and model, not just one invoice. Our guide to tracking Claude and GPT token costs by developer, team and model shows how. If your local estimate and your invoice disagree, see why your Claude Code cost estimate doesn’t match your bill.

Frequently asked questions

Why is my prompt caching not working?

The most common causes are a changing prefix, a cache marker on content that varies every request, a prompt below the minimum length, and gaps longer than the cache lifetime. Check the usage fields in the response to see whether any tokens were read from cache.

How do I calculate prompt caching break-even?

Use k > (w − r) ÷ (1 − r), where w is the write multiplier, r is the read multiplier, and k is the number of requests within the cache lifetime. For a 5-minute Anthropic write and a 0.1x read, caching pays off from the second hit.

How do I check my prompt cache hit rate?

Divide cache read tokens by total input tokens. On Anthropic, total input is cache_read_input_tokens plus cache_creation_input_tokens plus input_tokens. On OpenAI, compare cached_tokens to prompt tokens.

Can prompt caching make my bill higher?

Yes, on providers that charge a premium for cache writes. If a cache rarely hits, you pay the write premium without the read discount. In the worked example above, an always-miss cache cost 25% more than no caching.

How long does a prompt cache last?

Anthropic’s default is 5 minutes, refreshed each time the cache is used, with a 1-hour option at higher write cost. OpenAI’s retention depends on the model and can be extended up to 24 hours in some cases.

What is the minimum prompt length for caching?

For Anthropic it ranges from 512 to 4,096 tokens depending on the model. For OpenAI it is 1,024 tokens. Shorter prompts aren’t cached and return no error.

Do cached tokens still count toward rate limits?

It depends on the provider. Anthropic’s docs say cache hits on the 1-hour cache aren’t deducted against your rate limit, and OpenAI’s FAQ says caching does not affect rate limits. Check each provider’s docs for your plan.