Asked about Attention, conceptually
Why did transformers replace recurrent networks?
An answer that works
Recurrence processes sequentially, so distant information degrades through a carried summary and nothing parallelises across the sequence. Attention connects any two positions directly and computes all positions at once, which is what made training at scale practical.
Reading an answer is not being able to give one. The section that teaches this is Attention, conceptually, 8 hours of Transformers & attention.
Learn this properlyAlso asked about Transformers & attention
Explain how attention works.Why is long context expensive?What do multiple attention heads do?What does causal masking do?What is the difference between attention and the feed-forward layer?Why does a transformer need position information at all?What are residual connections and layer norm for?What is the KV cache?
All 17 questions on Transformers & attention