Asked about From attention to a full language model

Why does a transformer need position information at all?

An answer that works

Attention compares every position with every other and blends the results, and none of those operations knows where anything is. With no position signal the model is exactly permutation-equivariant: shuffle the input tokens and the outputs come back in the same shuffled order, unchanged. Position has to be injected from outside, either as a vector added per slot or, in current models, by rotating queries and keys by an angle proportional to position so the comparison depends on the gap between two tokens.

Reading an answer is not being able to give one. The section that teaches this is From attention to a full language model, 6 hours of Transformers & attention.

Learn this properly

Also asked about Transformers & attention

Explain how attention works.Why did transformers replace recurrent networks?Why is long context expensive?What do multiple attention heads do?What does causal masking do?What is the difference between attention and the feed-forward layer?What are residual connections and layer norm for?What is the KV cache?
All 17 questions on Transformers & attention