Asked about Attention, conceptually
Explain how attention works.
An answer that works
Each position produces a query, a key and a value. The query is compared against every key to produce weights, and the output is a weighted mixture of the values. It lets any position draw directly on any other regardless of distance, and unlike recurrence it parallelises across the sequence.
Then they push further
Why divide by the square root of the key dimension?
Dot products grow with dimension, so without scaling the scores get large, softmax becomes extremely peaked and gradients vanish. Dividing by the square root of the dimension keeps the scores in a workable range.
What changes in a decoder-only model?
A causal mask stops each position attending to later ones. The model can then learn to predict the next token at every position at once during training, and generate one token at a time afterwards.
What does the KV cache store, and why?
The key and value vectors of tokens already processed. For each new token only its own query, key and value are computed and the past ones are reused. Generation gets much faster, at the cost of memory that grows with sequence length and batch size.
What gets marked up
Strong answers explain queries, keys and values in plain terms, the scaling, and the mask, and connect them to the cost of long context. Reciting the formula without explaining it is marked down.
Reading an answer is not being able to give one. The section that teaches this is Attention, conceptually, 8 hours of Transformers & attention.
Learn this properlyAlso asked about Transformers & attention