Speculative decoding

Also: draft-and-verify decoding

A small fast model drafts several tokens, and the large model checks them all in a single pass, keeping the longest correct prefix. Output is distributed exactly as the large model's would be; only latency changes.

A definition is not an understanding. This is taught properly in Making generation faster, part of Quantization & inference optimisation.

Learn this properly

Terms from the same topic

KV cachePrefillDecode phaseQLoRAQuantization