Asked about What makes inference expensive
Why is batching almost free?
An answer that works
The weights are read once and applied to every request in the batch, and reading them is the bottleneck. This is why hosted APIs are cheap and why self-hosting for low traffic is poor value.
Reading an answer is not being able to give one. The section that teaches this is What makes inference expensive, 6 hours of Quantization & inference optimisation.
Learn this properlyAlso asked about Quantization & inference optimisation
What is the bottleneck in LLM inference?What does quantization trade?When would you self-host a model?Will an 8 billion parameter model serve 16 concurrent users with 8k contexts on a 24 GB GPU?Why does weight-only quantization speed up generation?Why does prompt length change time to first token but barely change time per output token?How would you work out the fastest a model could possibly generate on a given GPU?Why does continuous batching raise throughput so much?
All 14 questions on Quantization & inference optimisation