Asked about Making generation faster
What is speculative decoding, and what does it cost?
An answer that works
A small draft model proposes several tokens and the large model verifies them in one pass, accepting the longest prefix it agrees with. It is exact: the output distribution is the large model's. It costs the draft model's compute and only pays off above an acceptance rate of roughly two thirds, and it helps least on a server already running large batches, where there is no spare compute to spend.
Reading an answer is not being able to give one. The section that teaches this is Making generation faster, 5 hours of Quantization & inference optimisation.
Learn this properlyAlso asked about Quantization & inference optimisation
What is the bottleneck in LLM inference?What does quantization trade?Why is batching almost free?When would you self-host a model?Will an 8 billion parameter model serve 16 concurrent users with 8k contexts on a 24 GB GPU?Why does weight-only quantization speed up generation?Why does prompt length change time to first token but barely change time per output token?How would you work out the fastest a model could possibly generate on a given GPU?
All 14 questions on Quantization & inference optimisation