Asked about From attention to a full language model
What does causal masking do?
An answer that works
It prevents each position attending to later positions, so the model learns left-to-right prediction. It also lets the full sequence be trained in parallel, since every position predicts its next token at once.
Reading an answer is not being able to give one. The section that teaches this is From attention to a full language model, 6 hours of Transformers & attention.
Learn this properlyAlso asked about Transformers & attention
Explain how attention works.Why did transformers replace recurrent networks?Why is long context expensive?What do multiple attention heads do?What is the difference between attention and the feed-forward layer?Why does a transformer need position information at all?What are residual connections and layer norm for?What is the KV cache?
All 17 questions on Transformers & attention