Asked about What modern models actually do differently
What is grouped-query attention and why does it matter?
An answer that works
Several query heads share one key and value head instead of each having its own. The KV cache shrinks by exactly that ratio, so a model with 32 query heads and 8 key and value heads has a quarter of the cache. Since the cache rather than the weights usually caps concurrency, that is four times the requests or four times the context for the same memory. Quality drops slightly, and more so for multi-query attention with a single shared pair, which is why eight is the usual compromise.
Then they push further
You need more concurrency still. What next?
Quantize the cache, cap context length, use an engine that allocates cache in pages so nothing is reserved and wasted, and consider a model with windowed attention layers, where the cache stops growing past the window.
Does it make the weights smaller too?
Only slightly. The key and value projection matrices shrink by the same ratio, but they are a small share of a layer's parameters. The cache is where the saving is.
What gets marked up
Strong answers give the ratio and connect it to concurrency rather than to speed. Saying it makes the model faster without mentioning the cache misses the point.
Reading an answer is not being able to give one. The section that teaches this is What modern models actually do differently, 6 hours of Transformers & attention.
Learn this properlyAlso asked about Transformers & attention