RMSNorm

Also: root mean square normalisation

Layer normalisation with the mean subtraction dropped: divide the vector by its root mean square and apply a learned scale. Slightly cheaper, works just as well, and so has replaced layer norm in most recent models.

A definition is not an understanding. This is taught properly in What modern models actually do differently, part of Transformers & attention.

Learn this properly

Terms from the same topic

Transformer Attention Self-attention Multi-head attention Query, key and value Causal mask Positional encoding RoPE