Asked about Making generation faster

How would you work out the fastest a model could possibly generate on a given GPU?

An answer that works

At batch size one, each token requires reading every weight, so divide the weight bytes by the memory bandwidth. An 8B model at 16-bit is 16 GB, and on a 2 TB/s card that is 8 ms per token, about 125 tokens per second, before the KV cache reads and any inefficiency. Quantizing to 4-bit cuts the bytes by four and raises the ceiling by roughly the same factor.

Then they push further

What changes at batch size 64?

The weight read is amortised across 64 sequences, so total throughput rises nearly linearly while each sequence's token rate holds up until the batch becomes compute-bound. What limits it first is usually KV cache memory, not arithmetic.

Reading an answer is not being able to give one. The section that teaches this is Making generation faster, 5 hours of Quantization & inference optimisation.

Learn this properly

Also asked about Quantization & inference optimisation

What is the bottleneck in LLM inference?What does quantization trade?Why is batching almost free?When would you self-host a model?Will an 8 billion parameter model serve 16 concurrent users with 8k contexts on a 24 GB GPU?Why does weight-only quantization speed up generation?Why does prompt length change time to first token but barely change time per output token?Why does continuous batching raise throughput so much?
All 14 questions on Quantization & inference optimisation