Attention

Also: attention mechanism

A layer in which every position pulls information from other positions, weighting each one by how relevant it is. The weights are computed from the input itself, so which positions matter changes with every sequence.

A definition is not an understanding. This is taught properly in Attention, conceptually, part of Transformers & attention.

Learn this properly

Terms from the same topic

TransformerSelf-attentionMulti-head attentionQuery, key and valueCausal maskPositional encodingRoPERMSNorm