Multi-head attention

Also: MHA

Several attention computations run in parallel over different learned projections of the same input, then concatenated and projected back. Each head can track a different kind of relationship.

What people get wrong

More heads does not mean more compute. The per-head width is the model dimension divided by the number of heads, so eight narrow heads cost roughly what one wide one would.

A definition is not an understanding. This is taught properly in Attention, conceptually, part of Transformers & attention.

Learn this properly

Terms from the same topic

Transformer Attention Self-attention Query, key and value Causal mask Positional encoding RoPE RMSNorm