Prompt caching

A model re-reads your entire prompt on every request. If the first twenty thousand tokens are the same every time — a system prompt, a tool schema, a document the conversation is about — you are paying to process identical text repeatedly. Prompt caching lets the provider keep the computed state for that prefix and skip the work, which is cheaper and faster. This section covers what is actually cached, the one rule that decides whether you get a hit, how to order a prompt so you do, the arithmetic of whether it pays, and how to check from the response rather than by hoping.

What is cached, and why it is cheaper#

What is stored is not the text but the model's internal state after reading it — the key and value tensors for those tokens, the same KV cache that makes generation fast. Because attention is causal, every token attends only to what came before it, so that state depends on the prefix alone and is valid for any continuation. That is the whole mechanism, and it explains the constraint that follows: the cache works from the start of the prompt forward, and only for an exact match.

The pricing follows the work. Writing the prefix into the cache costs somewhat more than processing it normally, because the state has to be stored as well as computed. Reading it back costs a small fraction, because nothing is recomputed. Providers differ in the exact multipliers and in how long an entry survives, so check the ones you are billed under, but the shape is the same everywhere.

Prefix matching, and what invalidates it#

The match is on an exact prefix, byte for byte, from the very first token. Change anything early and everything after it is a miss, regardless of how much of the rest is identical. This is the single fact that determines whether caching works for you, and it has one consequence people discover the expensive way.

prompt                                          cached?
[system 20k][doc][question A]                   write
[system 20k][doc][question B]                   hit on the 20k + doc
[system 20k + today's date][doc][question C]    MISS on everything
[question D][system 20k][doc]                   MISS on everything

Ordering a prompt for caching#

The rule follows directly: arrange the prompt from least variable to most. Everything identical across all requests first, then material shared within a session, then the part that changes every time.

1. tool definitions            identical for the lifetime of the deployment
2. system instructions         identical until you deploy a change
3. few-shot examples           identical, and often large
4. retrieved documents         identical within a conversation
5. conversation history        grows, but only at the end
6. the current user message    different every time
7. the current date, ids       different every time

That last point is what makes caching so effective for agents in particular. An agent loop appends a tool call and its result to the end of a growing message list, so every iteration is a pure extension of the previous prefix. Handled correctly, a twenty-step agent pays full price for its context once and a tenth of it nineteen times. Handled carelessly — by rewriting or re-summarising the history in place — it pays full price twenty times for the same tokens.

Working out whether it pays#

Caching is not free on the first request, so for genuinely one-off prompts it is a loss. The break-even arrives quickly, and the arithmetic is worth doing on your own numbers rather than assuming either way.

BASE, WRITE, READ = 1.0, 1.25, 0.10     # multipliers on a normal input token

def cost(prefix, suffix, requests, cached):
    """Total input cost in units of one normal input token."""
    if not cached:
        return requests * (prefix + suffix) * BASE
    first = prefix * WRITE + suffix * BASE
    rest = (requests - 1) * (prefix * READ + suffix * BASE)
    return first + rest

prefix, suffix = 20_000, 500
for n in (1, 2, 3, 10, 100, 1000):
    u, c = cost(prefix, suffix, n, False), cost(prefix, suffix, n, True)
    print(f"{n:>5} requests  uncached {u:>10,.0f}  cached {c:>9,.0f}  {1 - c/u:>7.1%}")

#     1 requests  uncached     20,500  cached    25,500   -24.4%   <- a loss
#     2 requests  uncached     41,000  cached    28,000    31.7%
#     3 requests  uncached     61,500  cached    30,500    50.4%
#    10 requests  uncached    205,000  cached    48,000    76.6%
#   100 requests  uncached  2,050,000  cached   273,000    86.7%
#  1000 requests  uncached 20,500,000  cached 2,523,000    87.7%

The number that decides it in practice is not the request count but whether the second request arrives before the entry expires. Cache lifetimes are short — commonly a few minutes, extendable at extra cost — so a prefix reused twenty times an hour caches beautifully and the same prefix used twice a day never hits at all. Work out your reuse rate within the lifetime, not in total.

Verifying it from usage data#

Never assume a cache is working. Every provider reports cache reads and writes in the usage block of the response, and those fields are the only evidence that matters — a prompt you believe is stable is often not, and the fields say so immediately.

usage:
  input_tokens                    512     paid at the normal rate
  cache_creation_input_tokens  20,000     written this request
  cache_read_input_tokens           0     read from cache

next request, if all went well:
  input_tokens                    498
  cache_creation_input_tokens       0
  cache_read_input_tokens      20,000     the prefix was reused

Log those three numbers per request from the start. It turns cache behaviour into something you can graph and alert on, and the usual discovery is that a deploy changed a system prompt, or that one code path assembles the prompt in a different order from the rest — both of which show up as a cache hit rate falling off a cliff, and neither of which shows up any other way.

Prefix caching is not free, and whether it saves you anything depends on your traffic rather than your prompt. Writing a prefix into the cache costs more than plain input, reading it back costs much less, and the entry expires after minutes without use. With steady traffic that is a large saving; with one request an hour it is a loss. This section is that arithmetic, and then caching at the other end — storing whole responses, where an exact key is safe and a semantic one can serve a confidently wrong answer to a question that merely resembled an earlier one.

Writes, reads and expiry#

Prefix caching is not free on every provider. On some, writing a prefix into the cache costs a little more than normal input, and reading it back costs much less. Cached prefixes also expire after a period without use, often minutes, and providers set a minimum prefix length. Look up the current terms for your provider; they change.

Does it pay? Work it out#

Illustrative prices per million input tokens (check your provider):
  normal input   $3.00
  cache write    $3.75
  cache read     $0.30

Prefix of 8,000 tokens, 1,000 requests in the cache lifetime:
  without cache  1,000 x 8,000 x $3.00 / 1M          = $24.00
  with cache     1 write:  8,000 x $3.75 / 1M        = $0.03
                 999 reads: 999 x 8,000 x $0.30 / 1M = $2.40
                                                total = about $2.43

With steady traffic it is a large saving. With one request every hour and a cache that expires in minutes, every request pays the write price and caching costs more than not caching. Traffic pattern decides.

The variable is time, not the request count#

The prompt-caching section works out break-even in requests and finds it at the second one. That answer is only right if the second request arrives before the entry expires, and whether it does is a property of your traffic rather than your prompt. Model the arrivals as a Poisson process with a mean gap and the chance that the previous request is still cached is one minus the exponential of minus the lifetime over the gap. That gives an expected cost per request you can put a number on.

import math

BASE, WRITE, READ = 1.0, 1.25, 0.10      # multipliers on a normal input token
TTL = 300                                # seconds an unused entry survives

def effective(gap, ttl=TTL):
    """Poisson arrivals: the chance the previous request is still cached."""
    p_hit = 1 - math.exp(-ttl / gap)
    return p_hit, p_hit * READ + (1 - p_hit) * WRITE

print(f"{'mean gap':>10}{'hit rate':>10}{'cost/req':>10}{'vs no cache':>13}")
for gap in (5, 30, 120, 300, 900, 1800, 3600):
    p, c = effective(gap)
    print(f"{gap:>9}s{p:>10.0%}{c:>10.2f}{1 - c / BASE:>12.0%}")

p_break = (WRITE - BASE) / (WRITE - READ)
gap_break = -TTL / math.log(1 - p_break)
print(f"break-even hit rate {p_break:.1%}, reached at a mean gap of {gap_break:.0f}s")

#   mean gap  hit rate  cost/req  vs no cache
#         5s      100%      0.10         90%
#        30s      100%      0.10         90%
#       120s       92%      0.19         81%
#       300s       63%      0.52         48%
#       900s       28%      0.92          8%
#      1800s       15%      1.07         -7%
#      3600s        8%      1.16        -16%
# break-even hit rate 21.7%, reached at a mean gap of 1224s

That break-even hit rate falls straight out of the multipliers rather than out of the traffic: a write costs a quarter more than plain input and a read costs a tenth of it, so a hit has to happen often enough to repay the extra quarter, which is 0.25 divided by 1.15, or about 22%. Substitute your provider's numbers and the shape holds. It also tells you what to do about a prefix that is used too rarely — batch the work so requests arrive together, or pay for a longer lifetime where the provider sells one, and compare that price against the 25% write premium you are paying repeatedly instead.

Caching the whole response#

If the exact same request arrives again, you can return the stored answer and skip the model entirely. This suits repeated, non-personal questions and deterministic pipelines. The key must include everything that affects the answer.

import hashlib, json

def cache_key(model, prompt_version, messages, settings, tenant_id):
    raw = json.dumps([model, prompt_version, messages, settings, tenant_id], sort_keys=True)
    return hashlib.sha256(raw.encode()).hexdigest()

a = {"temperature": 0.0, "max_tokens": 512}
b = {"max_tokens": 512, "temperature": 0.0}      # the same settings, built elsewhere
msgs = [{"role": "user", "content": "What is the refund window?"}]

print("same key with sort_keys:  ",
      cache_key("m-1", "v3", msgs, a, "acme")[:12],
      cache_key("m-1", "v3", msgs, b, "acme")[:12])
print("without sort_keys:        ",
      hashlib.sha256(json.dumps(a).encode()).hexdigest()[:12],
      hashlib.sha256(json.dumps(b).encode()).hexdigest()[:12])
print("different tenant:         ",
      cache_key("m-1", "v3", msgs, a, "acme")[:12],
      cache_key("m-1", "v3", msgs, a, "globex")[:12])

# same key with sort_keys:   5d721c1d80c1 5d721c1d80c1
# without sort_keys:         0f6fb34da972 a78cfe8d527c
# different tenant:          5d721c1d80c1 b19d2c710f32

• The model id, including the dated version. A new model returns a different answer to the same prompt.
• The prompt version, bumped on every deploy that touches the template. Otherwise yesterday's wording is served all week.
• Every message, in full, and the sampling settings. Temperature above zero means the stored answer is one sample of many, which is a decision to make deliberately.
• The tenant or user, whenever the answer could differ between them. This is the field whose absence becomes an incident rather than a bug.
• A version of the underlying data, if the answer depends on retrieved documents. Changing a document has to invalidate the answers built from it.

An exact cache is safe when the same key genuinely implies the same correct answer, and that is a claim about your application rather than about caching. It holds for a documentation assistant answering the same five hundred questions all day. It does not hold for anything that reads live state, anything personalised, or anything where yesterday's answer is stale rather than merely old, and for those cases a short time to live is not a fix but a smaller version of the same bug.

Semantic caching is riskier than it looks#

A semantic cache returns a stored answer when a new question is similar enough to an old one, judged by embeddings. The danger is that similar is not the same. 'Cancel my order' and 'do not cancel my order' are close in embedding space. Answers that depend on who is asking can leak between users.

That is easy to say and easy to dismiss, so measure it. Take five cached questions, five genuine paraphrases that should hit them, and six near-misses that must not — the annual plan rather than the monthly one, the paid tier rather than the free one, a negated version of the same sentence. Then sweep the threshold and count, at each one, how many paraphrases hit the right entry and how many near-misses hit anything at all.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

CACHED = {
    "How do I cancel my subscription?": "Open Billing and choose Cancel plan.",
    "What is the refund window on the monthly plan?": "Fourteen days, in full.",
    "How do I reset my password?": "Use Forgot password on the sign-in page.",
    "Is there a discount for annual billing?": "Yes, two months free.",
    "What is the rate limit on the free tier?": "Sixty requests per minute.",
}
PARAPHRASE = {
    "How can I cancel my subscription?": "How do I cancel my subscription?",
    "Where do I reset my password?": "How do I reset my password?",
    "What is the refund period for monthly plans?": "What is the refund window on the monthly plan?",
    "Do you give a discount if I pay annually?": "Is there a discount for annual billing?",
    "How many requests per minute does the free tier allow?": "What is the rate limit on the free tier?",
}
NEAR_MISS = [
    "What is the refund window on the annual plan?",
    "What is the rate limit on the paid tier?",
    "I do not want to cancel my subscription, how do I keep it?",
    "How do I reset another user's password?",
    "Is there a discount for annual billing for non-profits?",
    "How do I cancel my free trial before it becomes a subscription?",
]

keys = list(CACHED)
vec = TfidfVectorizer().fit(keys + list(PARAPHRASE) + NEAR_MISS)   # stands in for embeddings
K = vec.transform(keys)

def nearest(question):
    sims = cosine_similarity(vec.transform([question]), K)[0]
    i = int(sims.argmax())
    return keys[i], float(sims[i])

print(f"{'threshold':>10}{'true hits':>11}{'wrong hits':>12}{'misses':>8}")
for t in [0.30, 0.40, 0.50, 0.60, 0.70, 0.80, 0.90]:
    hits = sum(nearest(q)[1] >= t and nearest(q)[0] == want for q, want in PARAPHRASE.items())
    wrong = sum(nearest(q)[1] >= t for q in NEAR_MISS)
    print(f"{t:>10.2f}{hits:>11}{wrong:>12}{len(PARAPHRASE) - hits:>8}")

right = [nearest(q)[1] for q, want in PARAPHRASE.items() if nearest(q)[0] == want]
wrong = [nearest(q)[1] for q in NEAR_MISS]
print(f"lowest correct paraphrase {min(right):.2f}, highest near-miss {max(wrong):.2f}")

for q in sorted(NEAR_MISS, key=lambda q: -nearest(q)[1])[:3]:
    match, sim = nearest(q)
    print(f"{sim:.2f}  {q}")
    print(f"      served: {CACHED[match]}")

#  threshold  true hits  wrong hits  misses
#       0.30          3           6       2
#       0.40          3           6       2
#       0.50          3           6       2
#       0.60          2           4       3
#       0.70          2           3       3
#       0.80          0           3       5
#       0.90          0           0       5
# lowest correct paraphrase 0.13, highest near-miss 0.88
# 0.88  What is the refund window on the annual plan?
#       served: Fourteen days, in full.
# 0.86  What is the rate limit on the paid tier?
#       served: Sixty requests per minute.
# 0.82  Is there a discount for annual billing for non-profits?
#       served: Yes, two months free.

Read the three worst rows as answers rather than as numbers. A customer on the annual plan is told the refund window is fourteen days. A customer on the paid tier is told their rate limit is sixty requests a minute. Both answers are fluent, confident, cached and wrong, and neither raises an error anywhere in your system — which is the difference between a semantic cache and every other cache you have ever run.

• Use it only for questions whose answer does not depend on the user.
• Set the similarity threshold from an eval of near-miss pairs, not from intuition.
• Scope the cache by tenant, and expire entries when the underlying data changes.
• Build the near-miss set out of the entities your answers turn on — plan names, tiers, regions, dates — because those are the words a single embedding will not keep apart.
• Log every semantic hit with both questions and the similarity. It is the only record that lets you find out afterwards what was served and to whom.