Causal mask

Also: causal masking, autoregressive mask

A constraint that stops each position attending to anything after it, applied by forcing those scores to negative infinity before the softmax. It is what lets a model generate: during training, position t only ever sees what preceded it.

What people get wrong

Break it and training loss falls suspiciously fast, because the model is reading the answer. A loss far below what the task should allow is nearly always a broken mask.

A definition is not an understanding. This is taught properly in Attention, conceptually, part of Transformers & attention.

Learn this properly