Positional encoding

Also: position embedding

Information about where a token sits in the sequence, mixed into its representation. Attention treats its input as a set, so without it "dog bites man" and "man bites dog" are the same input.

A definition is not an understanding. This is taught properly in From attention to a full language model, part of Transformers & attention.

Learn this properly

Terms from the same topic

Transformer Attention Self-attention Multi-head attention Query, key and value Causal mask RoPE RMSNorm