Asked about Causal multi-head attention in NumPy
Walk through the shapes of multi-head attention for a batch of 2, length 6, d_model 16 and 4 heads.
An answer that works
Input (2, 6, 16). Each projection keeps (2, 6, 16). Reshape to (2, 6, 4, 4) and transpose to (2, 4, 6, 4), so heads sit in front of positions. Scores are (2, 4, 6, 6), one square matrix per batch item per head, and the mask (6, 6) broadcasts over the first two axes. Weights times values gives (2, 4, 6, 4); transpose back to (2, 6, 4, 4), reshape to (2, 6, 16), then the output projection leaves it there.
Reading an answer is not being able to give one. The section that teaches this is Causal multi-head attention in NumPy, 6 hours of Transformers & attention.
Learn this properlyAlso asked about Transformers & attention
Explain how attention works.Why did transformers replace recurrent networks?Why is long context expensive?What do multiple attention heads do?What does causal masking do?What is the difference between attention and the feed-forward layer?Why does a transformer need position information at all?What are residual connections and layer norm for?
All 17 questions on Transformers & attention