RoPE

Also: rotary position embedding, rotary embeddings

Position encoded by rotating query and key vectors through an angle proportional to their index, so an attention score depends only on the distance between the two positions. Standard in modern open models, and the reason their context can often be stretched after training.

A definition is not an understanding. This is taught properly in What modern models actually do differently, part of Transformers & attention.

Learn this properly

Terms from the same topic

Transformer Attention Self-attention Multi-head attention Query, key and value Causal mask Positional encoding RMSNorm