Asked about From attention to a full language model

What is the KV cache?

An answer that works

Stored keys and values for tokens already processed, so each generation step computes only the new position rather than recomputing the whole prefix. It makes generation practical and consumes significant memory on long contexts.

Reading an answer is not being able to give one. The section that teaches this is From attention to a full language model, 6 hours of Transformers & attention.

Learn this properly

Also asked about Transformers & attention

Explain how attention works.Why did transformers replace recurrent networks?Why is long context expensive?What do multiple attention heads do?What does causal masking do?What is the difference between attention and the feed-forward layer?Why does a transformer need position information at all?What are residual connections and layer norm for?

All 17 questions on Transformers & attention