What modern models actually do differently

Open the config of any open-weights model released recently and you will find a dozen names that no introduction to transformers mentions: grouped-query attention, RoPE scaling, SwiGLU, a router and an expert count. This section goes through them one at a time and says what each changes about memory, speed or quality, with the arithmetic where there is arithmetic. It is the difference between knowing how a transformer works and being able to read what someone is actually shipping.

The model in the paper is not the model you will serve#

The 2017 architecture is the one most explanations still teach: multi-head attention, learned absolute position embeddings, a GELU feed-forward layer, layer normalisation. Every one of those has been replaced in the models being served today. Interviewers ask for the replacements by name, because each is a decision with a cost attached rather than a detail.

Grouped-query attention#

Multi-head attention gives every query head its own key and value heads. Keys and values are what the KV cache stores, so the cache grows with the number of heads. Grouped-query attention keeps all the query heads and lets several of them share one key and value head. Multi-query attention is the same idea taken to a single shared pair for the whole layer.

32 query heads, 32 KV heads   multi-head        full cache
32 query heads, 8 KV heads    grouped-query     cache / 4
32 query heads, 1 KV head     multi-query       cache / 32

KV cache per token = 2 x layers x kv_heads x head_dim x bytes

32 layers, 8 KV heads, head_dim 128, 16-bit   ->  128 KiB per token
the same model with 32 KV heads               ->  512 KiB per token

On a served model the cache, not the weights, is usually what caps how many requests fit on a card. Dividing it by four divides the memory cost of concurrency by four, or buys four times the context length for the same memory. This is the single change that made long context affordable, and it is why an 8B model can serve dozens of 8k conversations on one GPU.

The quality cost is small and real, and it grows as sharing gets more aggressive. Multi-query attention loses more and was reported to be less stable to train, which is why grouping, commonly eight key and value heads, is the compromise nearly everyone settled on.

Sliding-window and local attention#

Restrict each position to the last w tokens instead of the whole prefix. Cost per layer becomes linear in sequence length rather than quadratic, and the cache only has to hold w tokens per layer no matter how long the conversation gets.

What it costs is direct reach. Information still travels further than the window, because each layer moves it another w positions, so a stack of L layers reaches about L times w. What is lost is precise retrieval: a single token far outside the window cannot be attended to directly, which is exactly what a long-context lookup needs. Models that use it usually interleave, putting a few full-attention layers among many windowed ones so at least some layers can see everything.

Rotary position embeddings#

Attention treats its input as a set, so position has to be supplied from outside. Early models added a learned vector per position. That has two problems: the table has a fixed number of rows, so there is no vector at all for a position beyond the training length, and it encodes where a token sits rather than how far apart two tokens are.

RoPE rotates each query and key vector by an angle proportional to its position index. The dot product of a rotated query with a rotated key then depends only on the difference between the two positions, so relative distance falls out of the arithmetic instead of being learned. Nothing is added to the values and no table has to be sized in advance.

Extending context afterwards means changing the rotation frequencies so the angles a longer sequence produces stay inside the range the model saw in training. Position interpolation scales them down uniformly; the NTK-aware and YaRN variants scale them unevenly across frequency bands, leaving the high frequencies that carry local order alone. All of them work to a point, and all of them cost some quality, most visibly on precise retrieval inside the stretched range.

SwiGLU and RMSNorm#

The feed-forward layer used to be an up-projection, an activation such as GELU, and a down-projection. SwiGLU does the up-projection twice, passes one copy through a swish activation and multiplies the two elementwise, so one path gates the other. That is three matrices where there were two, so the hidden width is cut to roughly two thirds to keep the parameter count level. The gate lets the layer suppress channels per token, and it measures slightly better for the same budget. There is no deeper story than that.

RMSNorm drops the mean subtraction from layer normalisation: divide the vector by its root mean square and apply a learned scale, with no shift. One less reduction over the vector, marginally less arithmetic, and no measurable quality cost, which is why it has become the default.

Mixture of experts#

The feed-forward layer is split into many separate experts, and a small router sends each token to one or two of them. The others do nothing for that token. The model gets far more parameters without the compute per token rising to match.

Dense model       8B total parameters      8B active per token
Sparse model    120B total parameters      5B active per token

compute per token    follows the active count
memory to serve      follows the total count

The other costs are load balancing, which needs an auxiliary term in the loss to stop the router sending nearly everything to a few favoured experts, and communication: when experts are spread across devices, every token's routing decision becomes an all-to-all exchange.

FlashAttention#

Standard attention computes the full matrix of scores for every pair of positions, writes it to GPU memory, reads it back for the softmax, writes it again, and reads it once more to multiply by the values. On a long sequence that matrix is enormous, and moving it is what takes the time rather than the multiplying.

FlashAttention never writes it. It walks over blocks of queries and keys small enough to sit in on-chip memory, keeps a running maximum and sum so the softmax can be accumulated block by block, and builds the output as it goes. The backward pass recomputes the blocks it needs instead of storing them.

It is exact. The output matches standard attention to floating-point noise, it performs slightly more arithmetic rather than less, and the entire win is memory traffic. It also makes attention memory linear in sequence length rather than quadratic, which is what let context windows grow at all. If it appears in a list beside sparse and linear attention as another approximation, that is the thing to correct.