Capacity planning and load testing for an LLM service

Before a feature launches someone has to say how much traffic it can take and what that will cost. This section is that arithmetic and the test that checks it: how many requests will be in flight at once, how many tokens per minute they add up to, and what your service does when the provider starts refusing calls. Model calls are slow, so a modest request rate puts a surprising number of requests in flight, and the limit you hit first is usually tokens rather than requests.

How many requests are in flight?#

Little's law:  concurrent requests = arrival rate x time each takes

Peak 20 requests per second, each taking 6 seconds (a streamed answer):
  20 x 6 = 120 requests in flight at once

Your server, your connection pools and your provider limits must all allow 120.

Little's law is exact, it needs no assumption about the distribution of anything, and it is the one piece of queueing theory worth carrying around. The reason it surprises people is that model calls are slow: at web-service latencies of thirty milliseconds, twenty requests a second is 0.6 in flight and nobody thinks about concurrency at all. At six seconds it is 120, and every pool, worker count and quota on the path has to be sized for that number rather than for the twenty.

What the law does not tell you is what happens near the limit, and that is where services actually fall over. Simulate it: Poisson arrivals, a fixed number of concurrency permits, lognormal service times averaging six seconds.

import heapq
import numpy as np

def simulate(rate, service_mean, slots, n=200_000, seed=0):
    """Arrivals are Poisson at 'rate' per second. Each call holds one of 'slots'
    permits for a random time averaging 'service_mean' seconds."""
    rng = np.random.default_rng(seed)
    arrivals = np.cumsum(rng.exponential(1 / rate, n))
    service = rng.lognormal(np.log(service_mean) - 0.5 * 0.6 ** 2, 0.6, n)
    free = [0.0] * slots                       # when each permit next becomes free
    heapq.heapify(free)
    waits = np.empty(n)
    for i, (t, s) in enumerate(zip(arrivals, service)):
        start = max(t, heapq.heappop(free))
        waits[i] = start - t
        heapq.heappush(free, start + s)
    return waits, waits + service

SLOTS, SERVICE = 120, 6.0
capacity = SLOTS / SERVICE
print(f"{SLOTS} slots, {SERVICE:.0f}s per call -> capacity {capacity:.1f} req/s")
print(f"{'req/s':>6} {'util':>6} {'mean wait':>10} {'p95 wait':>9} {'p95 total':>10}")
for rate in [10, 14, 16, 18, 19, 19.5]:
    w, t = simulate(rate, SERVICE, SLOTS)
    print(f"{rate:>6} {rate / capacity:>6.2f} {w.mean():>10.2f} "
          f"{np.percentile(w, 95):>9.2f} {np.percentile(t, 95):>10.2f}")

120 slots, 6s per call -> capacity 20.0 req/s
 req/s   util  mean wait  p95 wait  p95 total
    10   0.50       0.00      0.00      13.43
    14   0.70       0.00      0.00      13.43
    16   0.80       0.00      0.00      13.44
    18   0.90       0.07      0.54      13.51
    19   0.95       0.40      1.83      13.92
  19.5   0.97       1.17      4.07      14.99

The shape of that column is the thing to remember. Between fifty and eighty per cent utilisation the queue does not exist: a request that arrives finds a free permit, every time. At ninety per cent a tail appears, and at ninety-seven per cent the p95 wait is four seconds, which on top of a six-second call is a user-visible failure. A service running comfortably at eighty per cent has no warning that it is three-tenths of its capacity away from being unusable, and traffic does not grow in careful increments.

Then the result that changes how you build the thing. Run the same simulation at a fixed ninety per cent utilisation and vary only how many permits the pool has.

print(f"{'slots':>6} {'req/s':>7} {'mean wait':>10} {'p95 wait':>9} {'p99 wait':>9}")
for slots in [4, 12, 40, 120]:
    rate = 0.9 * slots / SERVICE
    w, _ = simulate(rate, SERVICE, slots)
    print(f"{slots:>6} {rate:>7.1f} {w.mean():>10.2f} "
          f"{np.percentile(w, 95):>9.2f} {np.percentile(w, 99):>9.2f}")

 slots   req/s  mean wait  p95 wait  p99 wait
     4     0.6       8.70     30.39     45.87
    12     1.8       2.39      9.56     14.55
    40     6.0       0.48      2.43      3.93
   120    18.0       0.07      0.54      1.01

At ninety per cent utilisation, four permits give a mean wait of 8.7 seconds — longer than the call itself — and 120 permits give 0.07 seconds. That is a factor of 124 from pooling alone, with no extra capacity anywhere. The reason is that a small pool has no slack: when two long calls land together there is nothing left to absorb the third, whereas in a large pool the variation averages out across permits.

The practical consequence is that per-worker concurrency limits are expensive. Ten processes with a semaphore of twelve each is not the same system as ten processes sharing a pool of 120, even though both admit 120 requests: the first behaves like the twelve-slot row and the second like the 120-slot row. Where you can, hold the limit in one shared place — a token bucket in Redis, a single gateway, one async process rather than ten sync workers. Where you cannot, size the per-worker limit generously and enforce the real ceiling at a layer that sees all the traffic.

Provider limits are in tokens too#

20 requests/s x 60 = 1,200 requests per minute
each uses 3,000 input + 500 output tokens
  input tokens per minute  = 3.6 million
  output tokens per minute = 600,000

Check both against your account's limits, per model, before launch.

Load test the real thing#

Use a load testing tool such as Locust or k6 against a staging deployment with realistic prompts, ramping up to above expected peak. Record time to first token, total time, throughput in tokens per second, error rate and the rate of throttling responses. Point it at a cheaper model or a mock for the first runs to avoid a large bill.

When the provider says slow down#

import random, time

class RateLimited(Exception):
    def __init__(self, retry_after=None):
        self.retry_after = retry_after

class ServiceBusy(Exception):
    pass

def call_with_backoff(fn, max_tries=5):
    for attempt in range(max_tries):
        try:
            return fn()
        except RateLimited as e:
            wait = e.retry_after or min(30, 2 ** attempt)
            time.sleep(wait * random.uniform(0.5, 1.5))   # jitter spreads the retries out
    raise ServiceBusy()

The two lines that matter are the exponential term and the multiplier on it. Without jitter, every client that was throttled at the same moment waits the same one second, retries at the same moment, and recreates exactly the burst that caused the throttling — then waits two seconds and does it again. The retries are synchronised by the outage itself. Here is what that costs, with five hundred clients throttled simultaneously against a provider that will accept sixty calls a second.

def herd(jitter, clients=500, capacity=60, rounds=6, seed=0):
    """All clients are throttled at t=0. Each retries after 2**attempt seconds,
    optionally jittered. A retry succeeds only if fewer than 'capacity' others
    land in the same one-second bucket."""
    rng = np.random.default_rng(seed)
    pending = np.arange(clients)
    attempt = np.zeros(clients, dtype=int)
    done = np.zeros(clients, dtype=bool)
    t = np.zeros(clients)
    for _ in range(rounds):
        wait = 2.0 ** attempt[pending]
        if jitter:
            wait = wait * rng.uniform(0.5, 1.5, len(pending))
        t[pending] = t[pending] + wait
        bucket = np.floor(t[pending]).astype(int)
        seen, ok = {}, np.zeros(len(pending), dtype=bool)
        for k in np.argsort(t[pending]):
            b = int(bucket[k])
            seen[b] = seen.get(b, 0) + 1
            if seen[b] <= capacity:
                ok[k] = True
        done[pending[ok]] = True
        attempt[pending] += 1
        pending = pending[~ok]
        if len(pending) == 0:
            break
    return done.mean(), t[done].max()

for jitter in [False, True]:
    frac, worst = herd(jitter)
    print(f"jitter={str(jitter):<5} succeeded {frac:.1%} within 6 retries, "
          f"last success at t={worst:.1f}s")

jitter=False succeeded 72.0% within 6 retries, last success at t=63.0s
jitter=True  succeeded 100.0% within 6 retries, last success at t=9.7s

Without jitter, 28 per cent of clients still have not got through after six retries, and the ones that do are waiting up to 63 seconds, because each round only the first sixty arrivals in the synchronised burst are served and the rest are pushed into the next doubling. With jitter every client is through within ten seconds. Nothing about the capacity changed; the retries simply stopped arriving together. Honour the provider's retry-after header when it sends one, and jitter that too.

Shed load on purpose#

When demand exceeds capacity, an unbounded queue turns into minute-long waits and timeouts for everyone. Cap the queue, reject early with a clear message, prioritise paying or interactive traffic, and degrade gracefully, for example to a smaller model.