Time per output token

Also: TPOT, inter-token latency

The gap between successive tokens once generation is under way. Combined with time to first token and the length of the answer, it fully predicts how long a response takes.

A definition is not an understanding. This is taught properly in Find out where the money goes, part of Cost & latency engineering.

Learn this properly

Terms from the same topic

LatencyTime to first token