Attention, conceptually

Every current language model is a transformer, and the part of a transformer that does the work is attention. This section explains it in plain language with no equations, because being able to say what attention does in five minutes is a genuinely separate skill from being able to implement it — the first is asked in interviews constantly and the second is asked rarely. The implementation, with the shapes and the arithmetic, is the from-scratch section later in this topic.

The problem#

To understand a word you often need another word far away. In the sentence about the trophy not fitting in the suitcase because it was too big, working out what "it" refers to means looking back at two candidate nouns and deciding which one the word "big" makes sensible. Change "big" to "small" and the answer flips to the other noun. Nothing local to the word "it" contains that information.

Older sequence models processed one step at a time and carried a summary forward, so distant information degraded. They also could not be parallelised across the sequence, because step ten could not begin until step nine finished. The sequence-models topic measures exactly how badly the first problem bites: in a plain recurrent network the gradient reaching fifty steps back is around 5e-19, which is not small but absent.

Every position looks at every position#

Attention lets each position look directly at all the others in the same sequence — self-attention — and decide how much each one matters for it. Distance costs nothing: the fiftieth word and the first are one step apart, exactly as the fiftieth and the forty-ninth are. And because every position can do this at the same time, the whole sequence is processed in parallel, which is what made training at scale possible.

The output for each position is a blend: a weighted average of information drawn from every position, where the weights are how relevant each one was judged to be. Those weights are not fixed. They are computed fresh for every position on every input, which is why the same word contributes differently in different sentences, and why a model can resolve "it" one way in one sentence and the other way when a single adjective changes.

Queries, keys and values#

A useful analogy is a lookup. Each position produces a query describing what it is looking for, a key describing what it offers, and a value which is the information it contributes if selected. Every query is compared against every key, the comparisons become the weights, and the weights blend the values.

query   "I am a pronoun, what noun do I refer to?"
key     "I am a singular concrete noun"
value   the information contributed if selected

match query against every key   ->  a weight per position
blend the values by those weights  ->  this position's output

That last point is the one worth holding on to. If a position had only one representation, what it offers to others and what it seeks from them would be forced to be the same thing, and they are not. The pronoun is seeking a noun and offering the fact that it is a pronoun. Separating query from key lets those be different, and separating value from both lets what gets passed on differ again from what made it match.

Multiple heads#

One set of weights captures one kind of relationship. Several heads run in parallel with different learned parameters, so one can track grammatical agreement while another tracks which noun a pronoun refers to and another tracks position in a list. Their outputs are combined, so the layer can express several relationships at once rather than being forced to choose.

It is tempting to think each head learns one clean, nameable job, and the honest position is that some do and most do not. Interpretability research has found heads with crisp roles — copying a token from earlier, tracking the previous token — and also found that heads share work, change role with context, and rarely divide up as neatly as the explanation suggests. The interpretability topic measures this directly: ablating heads one at a time leaves accuracy at 1.000 for six of the eight, and above 0.99 for the other two, while the logit margin moves by between 1.02 and 5.18, so the ones that look unused are doing something.

The cost#

Every position compares against every position, so doubling the length quadruples the comparisons. This is the single most important consequence of the design, and it is easier to believe as a table than as a sentence.

  tokens     comparisons    vs 1k    KV cache, 8B model
   1,000       1,000,000       1x               0.12 GB
   2,000       4,000,000       4x               0.24 GB
   8,000      64,000,000      64x               0.98 GB
  32,000   1,024,000,000    1024x               3.91 GB
 128,000  16,384,000,000   16384x              15.62 GB
1,000,000 1,000,000,000,000 1000000x           122.07 GB

Two things follow that come up constantly in practice. Long context is expensive in a way that is not linear, so a prompt twice as long is not twice the cost; and the memory that grows is the cache of keys and values for everything already read, which is why generation slows as a conversation lengthens even though each new token is one token. Most of the work in modern architectures — the variants in the next section — is about making one or both of those grow more slowly.

How deep to go#

For applied work this level is sufficient: what attention does, why it replaced recurrence, what the three roles are for, and what it costs. If you can say those in five minutes without writing anything down, you can hold your end of any conversation an AI engineering interview will have about transformers.

Go further in two cases. If you are interviewing for research or for a role that trains models, implement causal multi-head attention from scratch — the from-scratch section in this topic does it in NumPy, and the exercise is an afternoon rather than a week. And if you are debugging inference cost or latency, read the modern-variants section instead, because what you need is not the mechanism but what recent architectures changed about it and why.