Asked about A decode loop with and without a KV cache
How does prompt caching relate to the KV cache?
An answer that works
It is the same data, kept between requests rather than within one. A shared prefix produces identical keys and values every time it is processed, because nothing after it can affect them, so a server that has already prefilled that prefix can reuse those rows and skip the prefill work. That is why prompt caching is billed against input tokens and why it only helps when the prefix is genuinely identical from the first token.
Reading an answer is not being able to give one. The section that teaches this is A decode loop with and without a KV cache, 4 hours of Quantization & inference optimisation.
Learn this properlyAlso asked about Quantization & inference optimisation
What is the bottleneck in LLM inference?What does quantization trade?Why is batching almost free?When would you self-host a model?Will an 8 billion parameter model serve 16 concurrent users with 8k contexts on a 24 GB GPU?Why does weight-only quantization speed up generation?Why does prompt length change time to first token but barely change time per output token?How would you work out the fastest a model could possibly generate on a given GPU?
All 14 questions on Quantization & inference optimisation