Asked about Capacity planning and load testing for an LLM service
You expect 50 requests per second at peak, each taking about 8 seconds. What capacity do you need?
An answer that works
By Little's law about 400 concurrent requests. Every layer, from server workers and connection pools to the provider's concurrency, requests per minute and tokens per minute limits, must support that with headroom. Then I would load test to confirm and plan throttling behaviour.
Then they push further
The provider throttles you at peak. What does your service do?
Retry with exponential backoff and jitter, honouring retry-after, cap the internal queue, prioritise interactive requests, fall back to another model or provider if one is set up, and show users a clear message rather than hanging.
What gets marked up
Marked up for Little's law, token-based limits, and jittered backoff.
Reading an answer is not being able to give one. The section that teaches this is Capacity planning and load testing for an LLM service, 4 hours of Deployment & serving.
Learn this properlyAlso asked about Deployment & serving