Sliding-window attention

Also: local attention

Restricting each position to attend only to the last w tokens instead of the whole prefix, making cost linear in sequence length rather than quadratic.

What people get wrong

Information still travels further than the window: each layer moves it another w positions, so a deep stack has an effective reach of layers times window.

A definition is not an understanding. This is taught properly in What modern models actually do differently, part of Transformers & attention.

Learn this properly

Terms from the same topic

Transformer Attention Self-attention Multi-head attention Query, key and value Causal mask Positional encoding RoPE