Asked about From attention to a full language model

What is the difference between attention and the feed-forward layer?

An answer that works

Attention moves information between positions. The feed-forward layer transforms each position independently. A block alternates the two, with residual connections around each.

Then they push further

Which holds more parameters?

The feed-forward layers, by a long way. On a typical 8B configuration they are about 70 per cent of the model against 17 per cent for all of attention, or 176 million per layer against 42 million. Attention gets discussed more because its cost grows with the square of sequence length, not because it is where the weights are.

Reading an answer is not being able to give one. The section that teaches this is From attention to a full language model, 6 hours of Transformers & attention.

Learn this properly

Also asked about Transformers & attention

Explain how attention works.Why did transformers replace recurrent networks?Why is long context expensive?What do multiple attention heads do?What does causal masking do?Why does a transformer need position information at all?What are residual connections and layer norm for?What is the KV cache?

All 17 questions on Transformers & attention