Prompt caching interview questions
6 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Prompt caching
Read the section that teaches thisIf this comes up in an interview
What is prompt caching and when does it help?
Providers cache the processed form of a stable prompt prefix and charge much less to reuse it. It helps whenever calls share a long fixed preamble: a large system prompt, tool definitions, few-shot examples, or a document queried repeatedly.
What breaks prompt caching?
Any change in the prefix, since matching runs from the start and stops at the first difference. Timestamps, request ids, a user name near the top, or JSON serialised with unstable key order all invalidate it silently.
How do you verify caching is working?
Check the cached token count in usage data on repeated similar requests. Consistently zero means something volatile is in the prefix. It is worth asserting on in a test, because the failure has no behavioural symptom.
The cost maths, and caching whole responses
Read the section that teaches thisIf this comes up in an interview
When does prompt caching not save money?
When requests are too infrequent for the cache to stay warm, so most requests pay the write price, or when the prompt changes near the start so the prefix rarely matches. Work out the numbers from your real traffic before assuming a saving.
Then they push further
How would you confirm the saving in production?
Read the cached and uncached token counts the API reports for each request, compute the hit rate and the effective price per request, and compare with the month before.
What gets marked up. Strong answers reason from traffic pattern and prices, not from 'caching is always cheaper'.
What are the risks of semantic caching?
Returning an answer to a question that is similar but means something different, and leaking answers between users when responses are personal. Limit it to non-personal questions, tune the threshold on near-miss examples, and scope it by tenant.
Then they push further
What would be in your cache key for an exact response cache?
The model and version, the prompt version, the full messages, the sampling settings, and the tenant or user when answers can differ between them. Leaving one out means serving an answer produced under different conditions.
Your cached prefix has a five-minute lifetime. How often do requests have to arrive before caching pays?
Break-even is a hit rate, and it comes from the multipliers: a write costs about 25% more than plain input and a read a tenth of it, so a hit has to repay the extra quarter, which is 0.25 over 1.15, or roughly 22%. Treating arrivals as Poisson, that hit rate is reached when the mean gap is about four times the lifetime, so with five minutes you need a request roughly every twenty minutes. Rarer than that and caching costs more than not caching.
Then they push further
It is rarer than that and you cannot change the traffic. What now?
Make the requests arrive together rather than spread out, by batching the work, so one write serves several reads. Or buy a longer lifetime if the provider sells one and compare its price against the write premium you are paying repeatedly. Or accept that this prefix should not be cached and spend the effort on a prefix that is.
What gets marked up. Strong answers reason in hit rate against the lifetime rather than in total request count, and know that a cache can be a net loss rather than merely ineffective.
These are the questions for one topic. The whole question bank covers every topic on the site.