Asked about What modern models actually do differently
Is FlashAttention an approximation?
An answer that works
No. It computes exactly the same attention, tiling the work so blocks fit in on-chip memory and using a running softmax so the full scores matrix is never written to GPU memory. It does slightly more arithmetic and far less memory traffic, which is the whole speedup, and it makes attention memory linear in sequence length.
Reading an answer is not being able to give one. The section that teaches this is What modern models actually do differently, 6 hours of Transformers & attention.
Learn this properlyAlso asked about Transformers & attention
Explain how attention works.Why did transformers replace recurrent networks?Why is long context expensive?What do multiple attention heads do?What does causal masking do?What is the difference between attention and the feed-forward layer?Why does a transformer need position information at all?What are residual connections and layer norm for?
All 17 questions on Transformers & attention