Asked about A decode loop with and without a KV cache
A naive loop regenerates 500 tokens from a 2,000-token prompt. How much work is wasted?
An answer that works
Each step re-projects the whole prefix, so step t does work proportional to t rather than to one. Summed over 500 steps from a base of 2,000 that is roughly 1.1 million token-projections instead of 500, a factor of about two thousand. The attention scores are worse still: the naive loop builds the full t-by-t matrix each step and reads one row of it, which is cubic in total against the cached version's quadratic.
Reading an answer is not being able to give one. The section that teaches this is A decode loop with and without a KV cache, 4 hours of Quantization & inference optimisation.
Learn this properlyAlso asked about Quantization & inference optimisation
What is the bottleneck in LLM inference?What does quantization trade?Why is batching almost free?When would you self-host a model?Will an 8 billion parameter model serve 16 concurrent users with 8k contexts on a 24 GB GPU?Why does weight-only quantization speed up generation?Why does prompt length change time to first token but barely change time per output token?How would you work out the fastest a model could possibly generate on a given GPU?
All 14 questions on Quantization & inference optimisation