FlashAttention

An attention implementation that never materialises the full attention matrix in GPU memory, computing it in tiles inside fast on-chip memory instead. Several times quicker, and a large part of why long contexts are affordable.

What people get wrong

It is not an approximation. Results match standard attention to floating-point noise; the saving is memory traffic, not fewer operations.

A definition is not an understanding. This is taught properly in What modern models actually do differently, part of Transformers & attention.

Learn this properly

Terms from the same topic

Transformer Attention Self-attention Multi-head attention Query, key and value Causal mask Positional encoding RoPE