Asked about What modern models actually do differently
Why did rotary position embeddings replace learned absolute positions?
An answer that works
A learned table has a fixed number of rows, so there is nothing for a position beyond the training length, and it encodes absolute position when what attention needs is relative distance. RoPE rotates queries and keys by an angle proportional to position, so their dot product depends only on the gap between them. Extending context means rescaling those frequencies, which works to a point and costs some retrieval quality.
Reading an answer is not being able to give one. The section that teaches this is What modern models actually do differently, 6 hours of Transformers & attention.
Learn this properlyAlso asked about Transformers & attention
Explain how attention works.Why did transformers replace recurrent networks?Why is long context expensive?What do multiple attention heads do?What does causal masking do?What is the difference between attention and the feed-forward layer?Why does a transformer need position information at all?What are residual connections and layer norm for?
All 17 questions on Transformers & attention