Asked about Serving open models yourself
Why does an inference server like vLLM get much higher throughput than a simple generation loop?
An answer that works
Continuous batching keeps the GPU full by adding and removing sequences at every step, and paged attention allocates KV cache memory in blocks so many more sequences fit. Together they raise the number of concurrent requests and the tokens per second per GPU.
Then they push further
Throughput is great but users complain about slow first tokens at peak. What do you change?
Limit batch size or concurrent sequences per instance to protect latency, add instances and scale on queue depth, separate long-context requests, and consider a smaller or quantized model for interactive traffic.
What gets marked up
Marked up for continuous batching, paged KV cache, and treating latency and throughput as a trade to be set by a target.
Reading an answer is not being able to give one. The section that teaches this is Serving open models yourself, 4 hours of Deployment & serving.
Learn this properlyAlso asked about Deployment & serving