Prompt caching

A model re-reads your entire prompt on every request. If the first twenty thousand tokens are the same every time — a system prompt, a tool schema, a document the conversation is about — you are paying to process identical text repeatedly. Prompt caching lets the provider keep the computed state for that prefix and skip the work, which is cheaper and faster. This section covers what is actually cached, the one rule that decides whether you get a hit, how to order a prompt so you do, the arithmetic of whether it pays, and how to check from the response rather than by hoping.

What is cached, and why it is cheaper#

What is stored is not the text but the model's internal state after reading it — the key and value tensors for those tokens, the same KV cache that makes generation fast. Because attention is causal, every token attends only to what came before it, so that state depends on the prefix alone and is valid for any continuation. That is the whole mechanism, and it explains the constraint that follows: the cache works from the start of the prompt forward, and only for an exact match.

The pricing follows the work. Writing the prefix into the cache costs somewhat more than processing it normally, because the state has to be stored as well as computed. Reading it back costs a small fraction, because nothing is recomputed. Providers differ in the exact multipliers and in how long an entry survives, so check the ones you are billed under, but the shape is the same everywhere.

Prefix matching, and what invalidates it#

The match is on an exact prefix, byte for byte, from the very first token. Change anything early and everything after it is a miss, regardless of how much of the rest is identical. This is the single fact that determines whether caching works for you, and it has one consequence people discover the expensive way.

prompt                                          cached?
[system 20k][doc][question A]                   write
[system 20k][doc][question B]                   hit on the 20k + doc
[system 20k + today's date][doc][question C]    MISS on everything
[question D][system 20k][doc]                   MISS on everything

Ordering a prompt for caching#

The rule follows directly: arrange the prompt from least variable to most. Everything identical across all requests first, then material shared within a session, then the part that changes every time.

1. tool definitions            identical for the lifetime of the deployment
2. system instructions         identical until you deploy a change
3. few-shot examples           identical, and often large
4. retrieved documents         identical within a conversation
5. conversation history        grows, but only at the end
6. the current user message    different every time
7. the current date, ids       different every time

That last point is what makes caching so effective for agents in particular. An agent loop appends a tool call and its result to the end of a growing message list, so every iteration is a pure extension of the previous prefix. Handled correctly, a twenty-step agent pays full price for its context once and a tenth of it nineteen times. Handled carelessly — by rewriting or re-summarising the history in place — it pays full price twenty times for the same tokens.

Working out whether it pays#

Caching is not free on the first request, so for genuinely one-off prompts it is a loss. The break-even arrives quickly, and the arithmetic is worth doing on your own numbers rather than assuming either way.

BASE, WRITE, READ = 1.0, 1.25, 0.10     # multipliers on a normal input token

def cost(prefix, suffix, requests, cached):
    """Total input cost in units of one normal input token."""
    if not cached:
        return requests * (prefix + suffix) * BASE
    first = prefix * WRITE + suffix * BASE
    rest = (requests - 1) * (prefix * READ + suffix * BASE)
    return first + rest

prefix, suffix = 20_000, 500
for n in (1, 2, 3, 10, 100, 1000):
    u, c = cost(prefix, suffix, n, False), cost(prefix, suffix, n, True)
    print(f"{n:>5} requests  uncached {u:>10,.0f}  cached {c:>9,.0f}  {1 - c/u:>7.1%}")

#     1 requests  uncached     20,500  cached    25,500   -24.4%   <- a loss
#     2 requests  uncached     41,000  cached    28,000    31.7%
#     3 requests  uncached     61,500  cached    30,500    50.4%
#    10 requests  uncached    205,000  cached    48,000    76.6%
#   100 requests  uncached  2,050,000  cached   273,000    86.7%
#  1000 requests  uncached 20,500,000  cached 2,523,000    87.7%

The number that decides it in practice is not the request count but whether the second request arrives before the entry expires. Cache lifetimes are short — commonly a few minutes, extendable at extra cost — so a prefix reused twenty times an hour caches beautifully and the same prefix used twice a day never hits at all. Work out your reuse rate within the lifetime, not in total.

Verifying it from usage data#

Never assume a cache is working. Every provider reports cache reads and writes in the usage block of the response, and those fields are the only evidence that matters — a prompt you believe is stable is often not, and the fields say so immediately.

usage:
  input_tokens                    512     paid at the normal rate
  cache_creation_input_tokens  20,000     written this request
  cache_read_input_tokens           0     read from cache

next request, if all went well:
  input_tokens                    498
  cache_creation_input_tokens       0
  cache_read_input_tokens      20,000     the prefix was reused

Log those three numbers per request from the start. It turns cache behaviour into something you can graph and alert on, and the usual discovery is that a deploy changed a system prompt, or that one code path assembles the prompt in a different order from the rest — both of which show up as a cache hit rate falling off a cliff, and neither of which shows up any other way.