Latency and cost budgets
What a person notices, what an LLM call actually costs in time, and the arithmetic that turns hardware price and throughput into a price per million tokens.
What a human perceives
| Under 100 ms | instant — feels like direct manipulation | The budget for a keystroke, a hover, a local filter. No network round trip fits here reliably. |
|---|---|---|
| 100 ms to 1 s | noticeable, but thought is not interrupted | No feedback needed beyond the result arriving. This is the target for retrieval and for classification calls. |
| 1 to 10 s | attention held, but a progress signal is required | Where most generation lands. Streaming converts this from a wait into reading time, which is most of what streaming buys. |
| Over 10 s | attention is lost; the task needs to become asynchronous | Past this, design for leaving and coming back — a job, a notification, a partial result — rather than a spinner. |
| Reading speed | about 5-8 tokens per second | Roughly 200-300 words per minute of comfortable reading, at about 1.3 tokens per word. Any stream faster than about 10 tokens per second is already outpacing the reader. |
Where the time goes
| Time to first token (TTFT) | under 500 ms is good · under 1 s acceptable · over 2 s feels broken | Dominated by prefill, which scales with prompt length, plus queueing under load. Halving the prompt is the most reliable way to halve it. |
|---|---|---|
| Time per output token (TPOT) | typically 10-50 ms per token from hosted models | So 20-100 tokens per second per stream. Small models and fast serving stacks reach the top of that range; large models at long context sit at the bottom. |
| Total latency | TTFT + (output tokens × TPOT) | Output length is usually the largest term and the easiest to control. Asking for a shorter answer is a latency optimisation. |
| Prefill | compute-bound, processes the whole prompt in parallel | Attention cost in prefill grows with the square of sequence length, so a prompt four times longer is more than four times the prefill work. |
| Decode | memory-bandwidth-bound, one token at a time | Every token requires reading all the weights from memory. This is why bandwidth, not FLOPs, sets single-stream generation speed. |
| Prompt caching | cuts TTFT substantially on a repeated prefix, and its input cost by roughly an order of magnitude | Exact discounts and cache lifetimes vary by provider and change. The structural rule does not: put the stable content first and the varying content last. |
Cost arithmetic
| Cost per million tokens, self-hosted | (hourly price ÷ (tokens per second × 3,600)) × 1,000,000 | A $2/hour GPU sustaining 1,000 tokens per second in aggregate costs about $0.56 per million tokens — at full utilisation. |
|---|---|---|
| Adjust for utilisation | divide by the fraction of the hour the GPU is actually busy | At 50% utilisation the real figure doubles. Utilisation is where self-hosting cost estimates usually go wrong, and it always goes wrong in the same direction. |
| Cost per request | (input tokens × input price + output tokens × output price) ÷ 1,000,000 | Output tokens are typically priced several times higher than input. Verbose formats are a cost decision as well as a latency one. |
| Batch tiers | roughly half price, for work tolerating hours of delay | Widely offered and worth restructuring for. Evaluation runs, backfills and bulk extraction almost all qualify. |
| Throughput versus latency | larger serving batches raise tokens per second per GPU and raise TPOT per stream | The same knob in both directions. Decide which one your product is actually paying for before tuning it. |
Figures last checked 2026-09-13. Several of these age; check anything you are about to rely on.
Where these numbers come from