Throughput
How much work a system completes per unit of time — tokens per second, requests per minute. The number that decides cost per unit served.
What people get wrong
Throughput and latency trade against each other. Larger batches raise total tokens per second while making each individual request slower.
A definition is not an understanding. This is taught properly in Capacity planning and load testing for an LLM service, part of Deployment & serving.
Learn this properlyTerms from the same topic