Mixture of experts

Also: MoE, sparse mixture of experts

A feed-forward layer split into many separate sub-networks, with a router sending each token to only one or two of them. Total parameters rise sharply while compute per token stays flat.

What people get wrong

Memory does not get cheaper. Every expert has to be resident even though few run per token, so a sparse model still needs the VRAM of its full parameter count.

A definition is not an understanding. This is taught properly in What modern models actually do differently, part of Transformers & attention.

Learn this properly

Terms from the same topic

Transformer Attention Self-attention Multi-head attention Query, key and value Causal mask Positional encoding RoPE