Asked about Causal multi-head attention in NumPy
Why does the softmax need the maximum subtracted?
An answer that works
Purely for numerical stability. Exponentiating a large score overflows to infinity in floating point and the division then gives NaN. Subtracting the row maximum is algebraically a no-op because the constant factors out of numerator and denominator, and it guarantees the largest exponent is exp(0), so nothing overflows and the smallest underflows harmlessly to zero.
Reading an answer is not being able to give one. The section that teaches this is Causal multi-head attention in NumPy, 6 hours of Transformers & attention.
Learn this properlyAlso asked about Transformers & attention
Explain how attention works.Why did transformers replace recurrent networks?Why is long context expensive?What do multiple attention heads do?What does causal masking do?What is the difference between attention and the feed-forward layer?Why does a transformer need position information at all?What are residual connections and layer norm for?
All 17 questions on Transformers & attention