Asked about A decode loop with and without a KV cache
Why is the query not cached alongside the key and value?
An answer that works
Because it is used exactly once. The query for position t is compared against every key up to t to produce that position's output, and no later position ever refers to it — later positions bring their own queries. Keys and values, by contrast, are read again at every subsequent step, which is what makes them worth storing.
Reading an answer is not being able to give one. The section that teaches this is A decode loop with and without a KV cache, 4 hours of Quantization & inference optimisation.
Learn this properlyAlso asked about Quantization & inference optimisation
What is the bottleneck in LLM inference?What does quantization trade?Why is batching almost free?When would you self-host a model?Will an 8 billion parameter model serve 16 concurrent users with 8k contexts on a 24 GB GPU?Why does weight-only quantization speed up generation?Why does prompt length change time to first token but barely change time per output token?How would you work out the fastest a model could possibly generate on a given GPU?
All 14 questions on Quantization & inference optimisation