Asked about Making generation faster

Why does prompt length change time to first token but barely change time per output token?

An answer that works

They are different workloads. Prefill processes the whole prompt in parallel, so it is compute-bound and scales with prompt length, and it is what time to first token measures. Each decode step reads the whole model to produce one token, so it is memory-bound and its cost is dominated by weight bytes over bandwidth, which does not change with prompt length. Only the growing KV cache adds a little.

Then they push further

Which one does your bill depend on?

Depends on the traffic. Long prompts with short answers, such as classification or extraction, are dominated by prefill and by input token pricing. Short prompts with long answers are dominated by decode, which is the slower and more expensive side per token.

How would you cut time to first token specifically?

Shorten the prompt, cache the shared prefix so it is not re-prefilled on every call, stream so the user sees output as it arrives, and keep long-context requests off the instances serving interactive traffic.

What gets marked up

Strong answers name the two phases, say which is compute-bound and which memory-bound, and then use that split to explain a serving decision.

Reading an answer is not being able to give one. The section that teaches this is Making generation faster, 5 hours of Quantization & inference optimisation.

Learn this properly

Also asked about Quantization & inference optimisation

What is the bottleneck in LLM inference?What does quantization trade?Why is batching almost free?When would you self-host a model?Will an 8 billion parameter model serve 16 concurrent users with 8k contexts on a 24 GB GPU?Why does weight-only quantization speed up generation?How would you work out the fastest a model could possibly generate on a given GPU?Why does continuous batching raise throughput so much?
All 14 questions on Quantization & inference optimisation