Asked about Causal multi-head attention in NumPy
Your attention weights sum to 0.6 on the first row and 1.0 on the last. What is wrong?
An answer that works
The mask is being applied after the softmax rather than before. The first position can see one entry out of many, so once the remaining entries are zeroed most of its normalised weight has been thrown away; the last position can see everything, so nothing is lost. Move the mask to the scores and use negative infinity.
Reading an answer is not being able to give one. The section that teaches this is Causal multi-head attention in NumPy, 6 hours of Transformers & attention.
Learn this properlyAlso asked about Transformers & attention
Explain how attention works.Why did transformers replace recurrent networks?Why is long context expensive?What do multiple attention heads do?What does causal masking do?What is the difference between attention and the feed-forward layer?Why does a transformer need position information at all?What are residual connections and layer norm for?
All 17 questions on Transformers & attention