Asked about Memory arithmetic and quantizing a model yourself
Why does weight-only quantization speed up generation?
An answer that works
Generating each token requires reading all the weights from memory, and that reading is the bottleneck. Smaller weights mean less to read per token, so each step is faster even if the arithmetic is done at higher precision.
Reading an answer is not being able to give one. The section that teaches this is Memory arithmetic and quantizing a model yourself, 4 hours of Quantization & inference optimisation.
Learn this properlyAlso asked about Quantization & inference optimisation
What is the bottleneck in LLM inference?What does quantization trade?Why is batching almost free?When would you self-host a model?Will an 8 billion parameter model serve 16 concurrent users with 8k contexts on a 24 GB GPU?Why does prompt length change time to first token but barely change time per output token?How would you work out the fastest a model could possibly generate on a given GPU?Why does continuous batching raise throughput so much?
All 14 questions on Quantization & inference optimisation