Deployment & serving

An AI service is an ordinary web service with a slow, expensive, unreliable dependency in the middle of it, and that one difference changes how you size it, how you time it out and what you watch. This section covers running one: how many requests to keep in flight when the ceiling belongs to your provider rather than your hardware, what to serve when the provider is down, and what belongs on the dashboard. It prevents the common outcome, which is a service that works fine in testing and returns an error to everyone the first time the upstream is slow.

Different constraints#

• Requests take seconds rather than milliseconds, so connection and worker limits behave differently.
• Concurrency is capped by the provider's rate limit, not by your CPU. A bigger machine does not help.
• Streaming means long-lived connections, which affects load balancers and timeouts.
• Cost varies per request, so a traffic spike is a budget event as well as a capacity one.
• The upstream dependency can be slow or down, and you have no control over it.

Three request shapes, and how to choose#

Before any sizing, decide what shape the request is, because it determines everything else: how long a connection is held, what the load balancer has to tolerate, and what the user sees while waiting. There are three, and the choice is made by how long the work takes rather than by preference.

shape          holds the connection   right when                    what it needs
synchronous    for the whole call     the answer takes under a      an ordinary timeout,
                                      few seconds and is short      generous but finite

streaming      for the whole call,    the answer is long and a      a load balancer that
               sending as it goes     person is reading it          does not buffer, and
                                                                    idle timeouts above
                                                                    the longest answer

asynchronous   not at all: returns    the work takes more than      a queue, a job store,
(job)          a job id immediately   about 30 seconds, or is       and a way to poll or
                                      an agent loop, or a batch     be called back

The mistake worth naming is running an agent loop, or anything that makes several model calls in sequence, over a synchronous request. Individually each call is fine; five of them in a row is ninety seconds, and somewhere between your client and your service is a proxy with a sixty-second idle timeout that nobody configured and nobody will find quickly. The symptom is a request that fails for the user while the work completes successfully on the server, which is the most expensive failure mode available: you pay for the tokens and deliver nothing. If the work can exceed about thirty seconds, return a job id.

Streaming sits in between, and its value is that it converts one long wait into an immediate first token, which changes what a user will tolerate without changing the total time or the cost at all. The price is that every layer between you and the browser must pass bytes through rather than buffering a response, and that a streamed failure arrives halfway through an answer rather than as a status code, so partial output has to be handled by the client.

Sizing concurrency#

The limit that matters is the provider's. Set your concurrency below it, share it deliberately across workers, and queue rather than rejecting when you hit it. One semaphore held for the duration of the call does this, and the reason it belongs at module scope rather than inside the handler is that a semaphore created per request limits nothing.

import asyncio, os, random, time

os.environ.setdefault("MAX_INFLIGHT", "8")
# Shared across the process, sized from the provider quota
MODEL_CONCURRENCY = asyncio.Semaphore(int(os.environ["MAX_INFLIGHT"]))

inflight = peak = 0

async def fake_provider(prompt):
    """Stands in for the model call: slow, and variable."""
    global inflight, peak
    inflight += 1
    peak = max(peak, inflight)
    try:
        await asyncio.sleep(random.uniform(0.02, 0.06))
        return f"answer to {prompt}"
    finally:
        inflight -= 1

async def generate(prompt):
    async with MODEL_CONCURRENCY:
        async with asyncio.timeout(90):
            return await fake_provider(prompt)

async def main():
    t = time.perf_counter()
    await asyncio.gather(*(generate(i) for i in range(200)))
    print(f"200 requests, peak concurrency {peak}, wall clock {time.perf_counter() - t:.2f}s")

asyncio.run(main())
# 200 requests, peak concurrency 8, wall clock ~1.0s  (timing varies per run)

Degrade rather than fail#

When the upstream is slow or unavailable, returning an error to every user is the worst outcome. Decide the fallback in advance.

1. 1.Retry briefly, with backoff and jitter.
2. 2.Fall back to a smaller or alternative model.
3. 3.Serve a cached or pre-computed answer if one is reasonable.
4. 4.Return a clear, honest message saying the feature is temporarily unavailable.

What to monitor#

Latency          p50 and p95, split by step so you can see which is slow
Error rate       by type: rate limited, timeout, upstream, your own
Cost             per request and per hour, with an alert on the rate
Token usage      input, output and cached, to see caching working
Queue depth      how many requests are waiting for a slot
Upstream health  the provider's status, since it is a dependency

Cost belongs on the dashboard next to latency. It is the metric most likely to move sharply after an innocent-looking change, and the one nobody notices until the invoice.

What the serving path actually costs#

Latency arguments go badly when they are conducted in adjectives. The fix is a budget: every component on the path, its median and its tail, written down before anyone optimises anything. Here is one for a service that classifies an incoming request using a small model and two feature lookups, with a 150 ms target at the 99th percentile. The per-component figures are the ones you measure on your own stack; what follows is what they add up to.

budget = [                 # component,                        p50 ms, p99 ms
    ("TLS + load balancer",                           4,     15),
    ("auth and request validation",                   2,      8),
    ("feature lookup: 3 keys from the online store", 12,     90),
    ("feature lookup: 1 aggregate from Postgres",    18,    140),
    ("preprocessing in Python",                       6,     20),
    ("model forward pass",                            9,     25),
    ("post-processing and response",                  3,     10),
]
w = max(len(c) for c, _, _ in budget)
for c, p50, p99 in budget:
    print(f"{c:<{w}} {p50:>6} {p99:>6}")
s50 = sum(p50 for _, p50, _ in budget)
s99 = sum(p99 for _, _, p99 in budget)
print(f"{'sum of components':<{w}} {s50:>6} {s99:>6}")
print(f"sum of p99s is {s99} ms, {s99 / 150:.1f}x a 150 ms budget")
print(f"the two feature lookups are {(90 + 140) / s99:.0%} of it; the model is {25 / s99:.0%}")

component                                       p50    p99
TLS + load balancer                               4     15
auth and request validation                       2      8
feature lookup: 3 keys from the online store     12     90
feature lookup: 1 aggregate from Postgres        18    140
preprocessing in Python                           6     20
model forward pass                                9     25
post-processing and response                      3     10
sum of components                                54    308

sum of p99s is 308 ms, 2.1x a 150 ms budget
the two feature lookups are 75% of it; the model is 8%

That is already the most useful output of the exercise, and it costs an afternoon of measurement. The model forward pass is eight per cent of the tail budget, so making the model twice as fast buys four per cent, and quantising it or moving it to a GPU buys a little more than nothing. Three quarters of the tail is two database calls.

Adding p99s is also not right, and it is worth knowing which direction it is wrong in. Components rarely peak together, so the sum of the p99s is an upper bound rather than the end-to-end p99. Simulating gives the real number, using a lognormal per component fitted so its median is the measured p50 and its 99th percentile is the measured p99.

import numpy as np

rng = np.random.default_rng(0)
n = 400_000

def simulate(components):
    total = np.zeros(n)
    for _, p50, p99 in components:
        sigma = np.log(p99 / p50) / 2.326          # 2.326 is the 99th percentile of a normal
        total += rng.lognormal(np.log(p50), sigma, n)
    return total

t = simulate(budget)
print(f"end to end: p50 {np.percentile(t, 50):.0f} ms, p95 {np.percentile(t, 95):.0f} ms, "
      f"p99 {np.percentile(t, 99):.0f} ms")

merged = [b for b in budget if not b[0].startswith("feature lookup")]
merged.append(("one batched feature lookup", 14, 95))
t2 = simulate(merged)
print(f"after merging the lookups: p50 {np.percentile(t2, 50):.0f} ms, "
      f"p95 {np.percentile(t2, 95):.0f} ms, p99 {np.percentile(t2, 99):.0f} ms")

end to end: p50 63 ms, p95 134 ms, p99 200 ms
after merging the lookups: p50 42 ms, p95 83 ms, p99 124 ms

So the honest end-to-end p99 is 200 ms against a 308 ms sum, and the service is over its budget by a third rather than by a factor of two. Merging the two feature lookups into one batched call — one round trip, one connection, the aggregate precomputed into the same store — takes the p99 to 124 ms and lands the service inside its target. No model work, no new hardware, one query.

Three habits come out of this and they transfer to any serving path. Write the budget before optimising, because the component you would have guessed is almost never the one. Never sum tail percentiles and call it a latency; either simulate or measure end to end. And count round trips before counting milliseconds, because on a path like this the number of sequential network hops predicts the tail better than any single component's speed.

Shipping the service once is the easy part. This section is the year afterwards: releasing a prompt change with the care a code change gets, since it can alter behaviour just as much and usually gets far less review; running an incident when the provider is degraded rather than down; planning capacity against rate limits you do not control; and alerting on cost per hour, so a loop that retries forever is caught the same afternoon rather than on the monthly invoice.

Prompt changes are deployments#

A prompt edit can change behaviour as much as a code change, and it is often made with far less care. Version prompts, run the eval suite before shipping them, and roll them out gradually like any other change.

Incidents#

1. 1.Is it us or the provider? Check the provider's status first.
2. 2.Did something change? Recent deploys, prompt edits, model version changes.
3. 3.What is the blast radius? All users, one tenant, one feature.
4. 4.Mitigate before diagnosing: roll back, switch to a fallback model, or disable the feature.
5. 5.Then find the cause, and write down what would have caught it earlier.

Capacity and cost#

Plan capacity against your provider rate limits, not your servers. Alert on cost per hour, not just total monthly spend, so a runaway process is caught within the hour rather than at the end of the month.

Shadow, canary, rollback#

Every change worth being careful about goes through the same three stages, and they answer different questions. Shadow answers whether the new thing produces different output, on real traffic, with nobody exposed to it. Canary answers whether the difference is bad, on a small share of real users. Rollback is what you do when the answer is yes, and it is the stage that gets designed last and used first.

The question nobody asks until it is too late is how long the canary has to run. It is not a judgement call; it is a sample size, and the sample size for a rate is set by the baseline and the size of the change you want to notice. Working it out takes one function, and the answer reframes the whole exercise.

import numpy as np
from scipy import stats

def n_needed(baseline, effect, power=0.8, alpha=0.05):
    """Requests per arm to notice a move from 'baseline' to 'baseline + effect'
    in a rate, at 80 per cent power and the usual 5 per cent significance."""
    p1, p2 = baseline, baseline + effect
    pbar = (p1 + p2) / 2
    z_a, z_b = stats.norm.ppf(1 - alpha / 2), stats.norm.ppf(power)
    num = (z_a * np.sqrt(2 * pbar * (1 - pbar))
           + z_b * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
    return int(np.ceil(num / effect ** 2))

CASES = [("error rate", 0.010, 0.005), ("error rate", 0.010, 0.002),
         ("conversion", 0.040, 0.004), ("fraud catch rate", 0.600, 0.030)]
print(f"{'metric':<18} {'base':>6} {'detect':>7} {'n per arm':>11}"
      f" {'5% of 15/s':>12} {'5% of 200/s':>13}")
for name, base, eff in CASES:
    n = n_needed(base, eff)
    print(f"{name:<18} {base:>6.3f} {eff:>7.3f} {n:>11,}"
          f" {n / (15 * 0.05) / 3600:>11.1f}h {n / (200 * 0.05) / 3600:>12.1f}h")

metric               base  detect   n per arm   5% of 15/s   5% of 200/s
error rate          0.010   0.005       7,750         2.9h          0.2h
error rate          0.010   0.002      42,693        15.8h          1.2h
conversion          0.040   0.004      39,475        14.6h          1.1h
fraud catch rate    0.600   0.030       4,129         1.5h          0.1h

A five per cent canary on a fifteen-requests-a-second service needs fifteen hours to notice a two-tenths-of-a-point move in the error rate, and fourteen hours to notice a small conversion change. The half-hour canary that everyone actually runs has nothing to say about either. What it does have something to say about is crashes, timeouts, latency and anything that goes wrong on nearly every request, because those need tens of samples rather than tens of thousands. So be explicit about which question the canary is answering: half an hour proves the thing does not fall over, and it is not evidence about a business metric.

• Size the canary before starting it, from the metric you intend to watch. If the required time is longer than you will wait, either raise the share of traffic or accept that you are only testing for crashes.
• Watch the guardrails continuously and the business metrics on their own timetable. A canary aborted at two minutes on a latency regression is a success, not a failed experiment.
• Route by a stable key, such as user id, rather than per request. A user who gets the old model on one request and the new one on the next sees the system contradict itself.
• Keep the canary at a fixed share while it runs. Ramping while measuring means the sample is drawn from a moving population, and the comparison no longer means what the arithmetic above says it means.

Rollback has one requirement that dominates all the others: it must be one action, performed by someone who did not build the system, at three in the morning, without a deploy. In practice that means the thing being changed is behind an indirection — a registry alias for a model, a versioned record for a prompt, a flag for a code path — so reverting is a value change rather than a release. The test of whether you have it is not whether the plan exists. It is whether anyone has done it this quarter.

The failures that never appear offline#

A class of production failure has the property that no offline evaluation can find it, because the thing that differs is not the model, the data or the code, but the circumstances of the call. Three of them account for most of what actually happens, and it is worth being able to name all three, because each has a different defence.

The first is feature skew: a feature computed one way in training and another way at serving. The MLOps platform section works through two measured cases, including the one where a normalisation refitted per batch is correct on a 20,000-row evaluation and wrong by a quarter of a probability on the online path, where the batch is one row. The defence is structural, not vigilance: one implementation of each feature, and preprocessing saved inside the model artefact.

The second is a stale feature store. The features exist, they are plausible, they are simply older than the request. Below, a model trained on features as of the scoring day is served features that are one to fourteen days old.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(9)
N, DAYS = 6000, 60
sessions = np.clip(rng.normal(4, 2, (N, DAYS)) + rng.normal(0, 1.5, (N, 1)), 0, None)
sessions = np.cumsum(sessions, axis=1) / np.arange(1, DAYS + 1)     # the slow-moving part
spike = rng.random((N, DAYS)) < 0.03
sessions = sessions + spike * rng.normal(6, 2, (N, DAYS))           # bursts: the fast part
tenure = rng.integers(1, 2000, N)

def outcome(day):
    logit = (0.45 * (sessions[:, day] - sessions[:, day].mean())
             - 0.0008 * (tenure - 1000) - 0.3)
    return (rng.random(N) < 1 / (1 + np.exp(-logit))).astype(int)

train_day = 30
clf = LogisticRegression(max_iter=1000).fit(
    np.column_stack([sessions[:, train_day], tenure]), outcome(train_day))

score_day = 55
y = outcome(score_day)
print(f"{'feature age':>12} {'AUC':>8} {'flagged':>9} {'precision':>11}")
for lag in [0, 1, 2, 3, 7, 14]:
    p = clf.predict_proba(np.column_stack([sessions[:, score_day - lag], tenure]))[:, 1]
    hit = p > 0.5
    print(f"{lag:>10}d  {roc_auc_score(y, p):>8.4f} {hit.mean():>9.3f} {y[hit].mean():>11.3f}")

 feature age      AUC   flagged   precision
         0d    0.7237     0.342       0.654
         1d    0.6943     0.343       0.624
         2d    0.6919     0.343       0.623
         3d    0.6909     0.347       0.620
         7d    0.6940     0.342       0.625
        14d    0.6884     0.346       0.623

Three points of AUC and three points of precision, lost to a cache that is one day behind, with nothing in any log to say so. The shape of the curve is the useful part: the loss is almost entirely between fresh and one day old, and fourteen days is barely worse than one. That is because only the fast-moving part of the feature carries the signal, and it is gone within a day. So the question to ask of any feature is not how stale the store is, it is which component of this feature moves on the timescale of the decision — and that is answerable offline, by exactly the table above, before the pipeline is built.

The third is a dependency upgrade changing preprocessing. A library bumps a default, a tokeniser normalises whitespace differently, an encoder changes its column order, and the model is fed something subtly unlike its training data. No offline evaluation catches this, because the evaluation runs on the new library too, so both sides move together and agree. The defence is a golden fixture: pin a handful of representative rows, hash the preprocessing output, and fail the build when the hash moves.

import hashlib, json
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

FIXTURE = pd.DataFrame({
    "tenure_days": [1, 30, 365, 1999],
    "plan": ["free", "pro", "team", "free"],
    "tickets_30d": [0, 2, 7, 1],
})

def build(drop=None):
    return ColumnTransformer([
        ("num", StandardScaler(), ["tenure_days", "tickets_30d"]),
        ("cat", OneHotEncoder(handle_unknown="ignore", drop=drop), ["plan"]),
    ])

def fingerprint(prep, df):
    out = np.asarray(prep.fit_transform(df), dtype=float)
    payload = json.dumps({"names": list(prep.get_feature_names_out()),
                          "values": np.round(out, 6).tolist()}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()[:16]

GOLDEN = fingerprint(build(), FIXTURE)
print("golden fingerprint", GOLDEN)

after = fingerprint(build(drop="first"), FIXTURE)      # a default changed under you
print("after upgrade     ", after)
print("guard:", "PASS" if after == GOLDEN else "FAIL - preprocessing changed, do not deploy")

golden fingerprint 2f36ba754b4808cc
after upgrade      ade0a912e4f0f389
guard: FAIL - preprocessing changed, do not deploy

Commit the golden hash alongside the code and regenerate it deliberately, as a reviewed change, never by rerunning the generator when the test goes red. The moment someone is allowed to refresh the fixture to make the build pass, the test has been converted into a slower way of doing nothing.

What to skip#

• Skip the long canary for a change that cannot plausibly move a business metric, such as a dependency bump behind a golden-fixture test. Watch the guardrails for ten minutes and go.
• Skip elaborate multi-arm rollouts with two hundred requests a second. There is not enough traffic to resolve two arms, let alone four.
• Skip writing a rollback plan you will not rehearse. An unrehearsed plan and no plan fail identically, and the first one is more expensive because it was believed.
• Worth memorising: canary duration is a sample-size calculation, rollback must be one action without a deploy, and the three offline-invisible failures are skew, staleness and a changed dependency.

Before a feature launches someone has to say how much traffic it can take and what that will cost. This section is that arithmetic and the test that checks it: how many requests will be in flight at once, how many tokens per minute they add up to, and what your service does when the provider starts refusing calls. Model calls are slow, so a modest request rate puts a surprising number of requests in flight, and the limit you hit first is usually tokens rather than requests.

How many requests are in flight?#

Little's law:  concurrent requests = arrival rate x time each takes

Peak 20 requests per second, each taking 6 seconds (a streamed answer):
  20 x 6 = 120 requests in flight at once

Your server, your connection pools and your provider limits must all allow 120.

Little's law is exact, it needs no assumption about the distribution of anything, and it is the one piece of queueing theory worth carrying around. The reason it surprises people is that model calls are slow: at web-service latencies of thirty milliseconds, twenty requests a second is 0.6 in flight and nobody thinks about concurrency at all. At six seconds it is 120, and every pool, worker count and quota on the path has to be sized for that number rather than for the twenty.

What the law does not tell you is what happens near the limit, and that is where services actually fall over. Simulate it: Poisson arrivals, a fixed number of concurrency permits, lognormal service times averaging six seconds.

import heapq
import numpy as np

def simulate(rate, service_mean, slots, n=200_000, seed=0):
    """Arrivals are Poisson at 'rate' per second. Each call holds one of 'slots'
    permits for a random time averaging 'service_mean' seconds."""
    rng = np.random.default_rng(seed)
    arrivals = np.cumsum(rng.exponential(1 / rate, n))
    service = rng.lognormal(np.log(service_mean) - 0.5 * 0.6 ** 2, 0.6, n)
    free = [0.0] * slots                       # when each permit next becomes free
    heapq.heapify(free)
    waits = np.empty(n)
    for i, (t, s) in enumerate(zip(arrivals, service)):
        start = max(t, heapq.heappop(free))
        waits[i] = start - t
        heapq.heappush(free, start + s)
    return waits, waits + service

SLOTS, SERVICE = 120, 6.0
capacity = SLOTS / SERVICE
print(f"{SLOTS} slots, {SERVICE:.0f}s per call -> capacity {capacity:.1f} req/s")
print(f"{'req/s':>6} {'util':>6} {'mean wait':>10} {'p95 wait':>9} {'p95 total':>10}")
for rate in [10, 14, 16, 18, 19, 19.5]:
    w, t = simulate(rate, SERVICE, SLOTS)
    print(f"{rate:>6} {rate / capacity:>6.2f} {w.mean():>10.2f} "
          f"{np.percentile(w, 95):>9.2f} {np.percentile(t, 95):>10.2f}")

120 slots, 6s per call -> capacity 20.0 req/s
 req/s   util  mean wait  p95 wait  p95 total
    10   0.50       0.00      0.00      13.43
    14   0.70       0.00      0.00      13.43
    16   0.80       0.00      0.00      13.44
    18   0.90       0.07      0.54      13.51
    19   0.95       0.40      1.83      13.92
  19.5   0.97       1.17      4.07      14.99

The shape of that column is the thing to remember. Between fifty and eighty per cent utilisation the queue does not exist: a request that arrives finds a free permit, every time. At ninety per cent a tail appears, and at ninety-seven per cent the p95 wait is four seconds, which on top of a six-second call is a user-visible failure. A service running comfortably at eighty per cent has no warning that it is three-tenths of its capacity away from being unusable, and traffic does not grow in careful increments.

Then the result that changes how you build the thing. Run the same simulation at a fixed ninety per cent utilisation and vary only how many permits the pool has.

print(f"{'slots':>6} {'req/s':>7} {'mean wait':>10} {'p95 wait':>9} {'p99 wait':>9}")
for slots in [4, 12, 40, 120]:
    rate = 0.9 * slots / SERVICE
    w, _ = simulate(rate, SERVICE, slots)
    print(f"{slots:>6} {rate:>7.1f} {w.mean():>10.2f} "
          f"{np.percentile(w, 95):>9.2f} {np.percentile(w, 99):>9.2f}")

 slots   req/s  mean wait  p95 wait  p99 wait
     4     0.6       8.70     30.39     45.87
    12     1.8       2.39      9.56     14.55
    40     6.0       0.48      2.43      3.93
   120    18.0       0.07      0.54      1.01

At ninety per cent utilisation, four permits give a mean wait of 8.7 seconds — longer than the call itself — and 120 permits give 0.07 seconds. That is a factor of 124 from pooling alone, with no extra capacity anywhere. The reason is that a small pool has no slack: when two long calls land together there is nothing left to absorb the third, whereas in a large pool the variation averages out across permits.

The practical consequence is that per-worker concurrency limits are expensive. Ten processes with a semaphore of twelve each is not the same system as ten processes sharing a pool of 120, even though both admit 120 requests: the first behaves like the twelve-slot row and the second like the 120-slot row. Where you can, hold the limit in one shared place — a token bucket in Redis, a single gateway, one async process rather than ten sync workers. Where you cannot, size the per-worker limit generously and enforce the real ceiling at a layer that sees all the traffic.

Provider limits are in tokens too#

20 requests/s x 60 = 1,200 requests per minute
each uses 3,000 input + 500 output tokens
  input tokens per minute  = 3.6 million
  output tokens per minute = 600,000

Check both against your account's limits, per model, before launch.

Load test the real thing#

Use a load testing tool such as Locust or k6 against a staging deployment with realistic prompts, ramping up to above expected peak. Record time to first token, total time, throughput in tokens per second, error rate and the rate of throttling responses. Point it at a cheaper model or a mock for the first runs to avoid a large bill.

When the provider says slow down#

import random, time

class RateLimited(Exception):
    def __init__(self, retry_after=None):
        self.retry_after = retry_after

class ServiceBusy(Exception):
    pass

def call_with_backoff(fn, max_tries=5):
    for attempt in range(max_tries):
        try:
            return fn()
        except RateLimited as e:
            wait = e.retry_after or min(30, 2 ** attempt)
            time.sleep(wait * random.uniform(0.5, 1.5))   # jitter spreads the retries out
    raise ServiceBusy()

The two lines that matter are the exponential term and the multiplier on it. Without jitter, every client that was throttled at the same moment waits the same one second, retries at the same moment, and recreates exactly the burst that caused the throttling — then waits two seconds and does it again. The retries are synchronised by the outage itself. Here is what that costs, with five hundred clients throttled simultaneously against a provider that will accept sixty calls a second.

def herd(jitter, clients=500, capacity=60, rounds=6, seed=0):
    """All clients are throttled at t=0. Each retries after 2**attempt seconds,
    optionally jittered. A retry succeeds only if fewer than 'capacity' others
    land in the same one-second bucket."""
    rng = np.random.default_rng(seed)
    pending = np.arange(clients)
    attempt = np.zeros(clients, dtype=int)
    done = np.zeros(clients, dtype=bool)
    t = np.zeros(clients)
    for _ in range(rounds):
        wait = 2.0 ** attempt[pending]
        if jitter:
            wait = wait * rng.uniform(0.5, 1.5, len(pending))
        t[pending] = t[pending] + wait
        bucket = np.floor(t[pending]).astype(int)
        seen, ok = {}, np.zeros(len(pending), dtype=bool)
        for k in np.argsort(t[pending]):
            b = int(bucket[k])
            seen[b] = seen.get(b, 0) + 1
            if seen[b] <= capacity:
                ok[k] = True
        done[pending[ok]] = True
        attempt[pending] += 1
        pending = pending[~ok]
        if len(pending) == 0:
            break
    return done.mean(), t[done].max()

for jitter in [False, True]:
    frac, worst = herd(jitter)
    print(f"jitter={str(jitter):<5} succeeded {frac:.1%} within 6 retries, "
          f"last success at t={worst:.1f}s")

jitter=False succeeded 72.0% within 6 retries, last success at t=63.0s
jitter=True  succeeded 100.0% within 6 retries, last success at t=9.7s

Without jitter, 28 per cent of clients still have not got through after six retries, and the ones that do are waiting up to 63 seconds, because each round only the first sixty arrivals in the synchronised burst are served and the rest are pushed into the next doubling. With jitter every client is through within ten seconds. Nothing about the capacity changed; the retries simply stopped arriving together. Honour the provider's retry-after header when it sends one, and jitter that too.

Shed load on purpose#

When demand exceeds capacity, an unbounded queue turns into minute-long waits and timeouts for everyone. Cap the queue, reject early with a clear message, prioritise paying or interactive traffic, and degrade gracefully, for example to a smaller model.

Running an open model on your own GPU means running an inference server, such as vLLM or SGLang, which loads the weights and exposes an API your existing client code can talk to almost unchanged. This section covers what those servers do to go fast, which is batching requests continuously and allocating cache memory in pages, how to work out what will fit on a given card, and the trade between total throughput and how quickly one user sees their first token. Without that last part you can tune a server to excellent benchmark numbers and a bad experience.

One command#

pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192
# an OpenAI-compatible API on port 8000

Inference engines such as vLLM and SGLang handle batching, memory management and streaming, and expose an API compatible with common client libraries, so application code barely changes when you switch from a hosted provider.

Why they are fast#

• Continuous batching: new requests join the running batch at each generation step instead of waiting for the batch to finish, which keeps the GPU busy.
• Paged KV cache, the trick vLLM calls PagedAttention: cache memory is allocated in small blocks as needed rather than reserved for the maximum length, so many more sequences fit at once.

Sizing#

Weights take parameters times bytes per parameter; a 7B model in 16-bit needs about 14 GB. Everything left over is KV cache, which decides how many concurrent sequences of a given length fit — and that number, not the weights, is what determines whether the server can carry your traffic. The cache holds a key and a value vector per layer per token, so the cost per token is two, times the layers, times the number of key-value heads, times the head dimension, times two bytes.

GIB = 1024 ** 3
MODELS = [                     # layers, kv heads, head dim, params
    ("Qwen2.5-7B",    28, 4, 128,  7.6e9),
    ("Llama-3.1-8B",  32, 8, 128,  8.0e9),
    ("Llama-3.1-70B", 80, 8, 128, 70.6e9),
]
for name, layers, kv_heads, head_dim, params in MODELS:
    per_tok = 2 * layers * kv_heads * head_dim * 2      # K and V, 2 bytes each
    weights = params * 2 / GIB                          # bf16
    print(f"{name:<14} weights {weights:6.1f} GiB   KV {per_tok / 1024:6.1f} KiB per token")
    for card in [24, 48, 80]:
        free = card * 0.92 - weights                    # 8% for activations and overhead
        if free <= 0:
            print(f"   {card:>3} GB card: does not fit")
            continue
        toks = free * GIB / per_tok
        print(f"   {card:>3} GB card: {free:5.1f} GiB free -> {toks:9,.0f} cached tokens"
              f" = {toks / 4096:5.0f} sequences at 4k, {toks / 32768:5.1f} at 32k")

Qwen2.5-7B     weights   14.2 GiB   KV   56.0 KiB per token
    24 GB card:   7.9 GiB free ->   148,372 cached tokens =    36 sequences at 4k,   4.5 at 32k
    48 GB card:  30.0 GiB free ->   561,810 cached tokens =   137 sequences at 4k,  17.1 at 32k
    80 GB card:  59.4 GiB free -> 1,113,061 cached tokens =   272 sequences at 4k,  34.0 at 32k
Llama-3.1-8B   weights   14.9 GiB   KV  128.0 KiB per token
    24 GB card:   7.2 GiB free ->    58,809 cached tokens =    14 sequences at 4k,   1.8 at 32k
    48 GB card:  29.3 GiB free ->   239,688 cached tokens =    59 sequences at 4k,   7.3 at 32k
    80 GB card:  58.7 GiB free ->   480,861 cached tokens =   117 sequences at 4k,  14.7 at 32k
Llama-3.1-70B  weights  131.5 GiB   KV  320.0 KiB per token
    24 GB card: does not fit
    48 GB card: does not fit
    80 GB card: does not fit

Three readings, and the second one is the surprise. First, the weights are not the constraint on a 24 GB card serving a 7B model — they take 14 GB and leave 8, and it is the 8 that decides whether you can serve fourteen users or thirty-six. Second, Qwen2.5-7B and Llama-3.1-8B are nearly the same size and their cache costs differ by a factor of 2.3, because Qwen uses four key-value heads and Llama uses eight. On the same 24 GB card that is 36 concurrent 4k sequences against 14. Grouped-query attention is a serving decision as much as an architecture one, and it is not visible in the parameter count anybody quotes.

Third, look at the 32k column. Concurrency at long context collapses: the 24 GB card carrying 36 short sequences carries four long ones. This is why a service that is comfortable on short prompts falls over the week someone ships a feature that pastes a document in, and why long-context and interactive traffic usually want separate pools rather than one queue. Treat these figures as an upper bound — the engine reserves memory for its own scheduling and fragments its pages — and then confirm with a load test.

The trade-off#

Bigger batches mean more tokens per second in total and more cost-efficient serving, but each user's tokens arrive a little slower and time to first token rises under load. The mechanism is worth being precise about, because it decides which knob to turn. Generating a token reads every weight in the model, so a batch of sixteen reads them once and does sixteen times the arithmetic: throughput rises nearly linearly with batch size while the cost per token falls. What rises with it is the time each individual user waits, because their token is now produced in a step that is doing sixteen times the work, and because a new request may wait for a scheduling slot before its prompt is even processed.

So the two numbers pull against each other and there is no setting that optimises both. Decide the latency target first — a time to first token a person will accept, usually somewhere under a second for interactive use — then find the highest concurrency that still meets it, and add instances beyond that rather than batch size. Reporting tokens per second without the latency it was measured at is the most common way these benchmarks mislead.

Scaling#

A new GPU instance must download and load the weights before serving, so a cold start can take minutes. Scale on queue depth or pending requests rather than CPU, keep a minimum number of warm instances, and cache model weights on fast local disk or in the image registry close to the machines. Scaling on CPU utilisation is the specific error: the CPU on an inference box is nearly idle by design, so the autoscaler sees a quiet machine while the queue is thirty deep.

What to skip#

• Skip self-hosting entirely until you have a reason a hosted API cannot meet: data that may not leave, a model nobody hosts, or a volume at which the arithmetic genuinely favours a rented card. An idle GPU costs the same as a busy one.
• Skip writing your own batching loop. Continuous batching and paged attention are the two things the engines do well and the two things that are hard to reproduce.
• Skip the tokens-per-second benchmark that was measured at a concurrency nobody will run at. Quote throughput at the latency you promised.
• Worth memorising: the KV formula, that grouped-query attention changes concurrency more than model size does, and that cold starts are minutes rather than seconds.