Multi-query attention

Also: MQA

Every query head sharing a single key and value head — grouped-query attention taken to its limit. The smallest possible cache, at some quality cost, which is why grouping is the usual compromise.

A definition is not an understanding. This is taught properly in What modern models actually do differently, part of Transformers & attention.

Learn this properly

Terms from the same topic

Transformer Attention Self-attention Multi-head attention Query, key and value Causal mask Positional encoding RoPE