Feed-forward network

Also: FFN, MLP block

The per-position half of a transformer block: expand to roughly four times the model dimension, apply a non-linearity, project back. Attention moves information between positions; this is where it gets processed.

A definition is not an understanding. This is taught properly in From attention to a full language model, part of Transformers & attention.

Learn this properly

Terms from the same topic

Transformer Attention Self-attention Multi-head attention Query, key and value Causal mask Positional encoding RoPE