SwiGLU

The feed-forward activation used in most current models: a gated linear unit with a swish activation, needing three weight matrices where a plain layer needs two. Slightly better loss per parameter than ReLU or GELU, at the same cost once widths are adjusted.

A definition is not an understanding. This is taught properly in What modern models actually do differently, part of Transformers & attention.

Learn this properly

Terms from the same topic

Transformer Attention Self-attention Multi-head attention Query, key and value Causal mask Positional encoding RoPE