Asked about What makes inference expensive
What does quantization trade?
An answer that works
Accuracy for memory and speed. 8 bit usually costs little, 4 bit costs something real and task-dependent. The loss is not uniform across tasks, so it has to be measured on your own eval rather than taken from a published aggregate.
Reading an answer is not being able to give one. The section that teaches this is What makes inference expensive, 6 hours of Quantization & inference optimisation.
Learn this properlyAlso asked about Quantization & inference optimisation
What is the bottleneck in LLM inference?Why is batching almost free?When would you self-host a model?Will an 8 billion parameter model serve 16 concurrent users with 8k contexts on a 24 GB GPU?Why does weight-only quantization speed up generation?Why does prompt length change time to first token but barely change time per output token?How would you work out the fastest a model could possibly generate on a given GPU?Why does continuous batching raise throughput so much?
All 14 questions on Quantization & inference optimisation