KV cache
Also: key-value cache
The stored keys and values for tokens already processed, so generating the next token does not recompute attention over the whole prefix. It turns generation from quadratic work into linear work.
What people get wrong
It is usually what limits how many requests fit on a GPU, not the weights: the cache grows with sequence length times batch size, and at long contexts it can exceed the model itself.
A definition is not an understanding. This is taught properly in What makes inference expensive, part of Quantization & inference optimisation.
Learn this properlyTerms from the same topic