Asked about What makes inference expensive
What is the bottleneck in LLM inference?
An answer that works
Memory bandwidth. Generating each token requires reading all the model weights from memory, so the GPU is mostly waiting on memory rather than computing. That is why quantization speeds things up and why batching is nearly free.
Then they push further
Why is generating tokens slower per token than reading the prompt?
The prompt is processed in parallel, all tokens at once, which keeps the GPU busy computing. Generation makes one token at a time, and each step reads every model weight from memory for very little arithmetic, so it is limited by memory bandwidth.
How does that explain why batching helps?
Each generation step reads the weights once whether it serves one request or fifty, so adding requests uses compute that would otherwise sit idle, until memory for the KV cache runs out.
What gets marked up
Marked up for separating prompt processing from generation, naming memory bandwidth, and linking it to batching and quantization.
Reading an answer is not being able to give one. The section that teaches this is What makes inference expensive, 6 hours of Quantization & inference optimisation.
Learn this properlyAlso asked about Quantization & inference optimisation