Running an AI service

An AI service is an ordinary web service with a slow, expensive, unreliable dependency in the middle of it, and that one difference changes how you size it, how you time it out and what you watch. This section covers running one: how many requests to keep in flight when the ceiling belongs to your provider rather than your hardware, what to serve when the provider is down, and what belongs on the dashboard. It prevents the common outcome, which is a service that works fine in testing and returns an error to everyone the first time the upstream is slow.

Different constraints#

• Requests take seconds rather than milliseconds, so connection and worker limits behave differently.
• Concurrency is capped by the provider's rate limit, not by your CPU. A bigger machine does not help.
• Streaming means long-lived connections, which affects load balancers and timeouts.
• Cost varies per request, so a traffic spike is a budget event as well as a capacity one.
• The upstream dependency can be slow or down, and you have no control over it.

Three request shapes, and how to choose#

Before any sizing, decide what shape the request is, because it determines everything else: how long a connection is held, what the load balancer has to tolerate, and what the user sees while waiting. There are three, and the choice is made by how long the work takes rather than by preference.

shape          holds the connection   right when                    what it needs
synchronous    for the whole call     the answer takes under a      an ordinary timeout,
                                      few seconds and is short      generous but finite

streaming      for the whole call,    the answer is long and a      a load balancer that
               sending as it goes     person is reading it          does not buffer, and
                                                                    idle timeouts above
                                                                    the longest answer

asynchronous   not at all: returns    the work takes more than      a queue, a job store,
(job)          a job id immediately   about 30 seconds, or is       and a way to poll or
                                      an agent loop, or a batch     be called back

The mistake worth naming is running an agent loop, or anything that makes several model calls in sequence, over a synchronous request. Individually each call is fine; five of them in a row is ninety seconds, and somewhere between your client and your service is a proxy with a sixty-second idle timeout that nobody configured and nobody will find quickly. The symptom is a request that fails for the user while the work completes successfully on the server, which is the most expensive failure mode available: you pay for the tokens and deliver nothing. If the work can exceed about thirty seconds, return a job id.

Streaming sits in between, and its value is that it converts one long wait into an immediate first token, which changes what a user will tolerate without changing the total time or the cost at all. The price is that every layer between you and the browser must pass bytes through rather than buffering a response, and that a streamed failure arrives halfway through an answer rather than as a status code, so partial output has to be handled by the client.

Sizing concurrency#

The limit that matters is the provider's. Set your concurrency below it, share it deliberately across workers, and queue rather than rejecting when you hit it. One semaphore held for the duration of the call does this, and the reason it belongs at module scope rather than inside the handler is that a semaphore created per request limits nothing.

import asyncio, os, random, time

os.environ.setdefault("MAX_INFLIGHT", "8")
# Shared across the process, sized from the provider quota
MODEL_CONCURRENCY = asyncio.Semaphore(int(os.environ["MAX_INFLIGHT"]))

inflight = peak = 0

async def fake_provider(prompt):
    """Stands in for the model call: slow, and variable."""
    global inflight, peak
    inflight += 1
    peak = max(peak, inflight)
    try:
        await asyncio.sleep(random.uniform(0.02, 0.06))
        return f"answer to {prompt}"
    finally:
        inflight -= 1

async def generate(prompt):
    async with MODEL_CONCURRENCY:
        async with asyncio.timeout(90):
            return await fake_provider(prompt)

async def main():
    t = time.perf_counter()
    await asyncio.gather(*(generate(i) for i in range(200)))
    print(f"200 requests, peak concurrency {peak}, wall clock {time.perf_counter() - t:.2f}s")

asyncio.run(main())
# 200 requests, peak concurrency 8, wall clock ~1.0s  (timing varies per run)

Degrade rather than fail#

When the upstream is slow or unavailable, returning an error to every user is the worst outcome. Decide the fallback in advance.

1. 1.Retry briefly, with backoff and jitter.
2. 2.Fall back to a smaller or alternative model.
3. 3.Serve a cached or pre-computed answer if one is reasonable.
4. 4.Return a clear, honest message saying the feature is temporarily unavailable.

What to monitor#

Latency          p50 and p95, split by step so you can see which is slow
Error rate       by type: rate limited, timeout, upstream, your own
Cost             per request and per hour, with an alert on the rate
Token usage      input, output and cached, to see caching working
Queue depth      how many requests are waiting for a slot
Upstream health  the provider's status, since it is a dependency

Cost belongs on the dashboard next to latency. It is the metric most likely to move sharply after an innocent-looking change, and the one nobody notices until the invoice.

What the serving path actually costs#

Latency arguments go badly when they are conducted in adjectives. The fix is a budget: every component on the path, its median and its tail, written down before anyone optimises anything. Here is one for a service that classifies an incoming request using a small model and two feature lookups, with a 150 ms target at the 99th percentile. The per-component figures are the ones you measure on your own stack; what follows is what they add up to.

budget = [                 # component,                        p50 ms, p99 ms
    ("TLS + load balancer",                           4,     15),
    ("auth and request validation",                   2,      8),
    ("feature lookup: 3 keys from the online store", 12,     90),
    ("feature lookup: 1 aggregate from Postgres",    18,    140),
    ("preprocessing in Python",                       6,     20),
    ("model forward pass",                            9,     25),
    ("post-processing and response",                  3,     10),
]
w = max(len(c) for c, _, _ in budget)
for c, p50, p99 in budget:
    print(f"{c:<{w}} {p50:>6} {p99:>6}")
s50 = sum(p50 for _, p50, _ in budget)
s99 = sum(p99 for _, _, p99 in budget)
print(f"{'sum of components':<{w}} {s50:>6} {s99:>6}")
print(f"sum of p99s is {s99} ms, {s99 / 150:.1f}x a 150 ms budget")
print(f"the two feature lookups are {(90 + 140) / s99:.0%} of it; the model is {25 / s99:.0%}")

component                                       p50    p99
TLS + load balancer                               4     15
auth and request validation                       2      8
feature lookup: 3 keys from the online store     12     90
feature lookup: 1 aggregate from Postgres        18    140
preprocessing in Python                           6     20
model forward pass                                9     25
post-processing and response                      3     10
sum of components                                54    308

sum of p99s is 308 ms, 2.1x a 150 ms budget
the two feature lookups are 75% of it; the model is 8%

That is already the most useful output of the exercise, and it costs an afternoon of measurement. The model forward pass is eight per cent of the tail budget, so making the model twice as fast buys four per cent, and quantising it or moving it to a GPU buys a little more than nothing. Three quarters of the tail is two database calls.

Adding p99s is also not right, and it is worth knowing which direction it is wrong in. Components rarely peak together, so the sum of the p99s is an upper bound rather than the end-to-end p99. Simulating gives the real number, using a lognormal per component fitted so its median is the measured p50 and its 99th percentile is the measured p99.

import numpy as np

rng = np.random.default_rng(0)
n = 400_000

def simulate(components):
    total = np.zeros(n)
    for _, p50, p99 in components:
        sigma = np.log(p99 / p50) / 2.326          # 2.326 is the 99th percentile of a normal
        total += rng.lognormal(np.log(p50), sigma, n)
    return total

t = simulate(budget)
print(f"end to end: p50 {np.percentile(t, 50):.0f} ms, p95 {np.percentile(t, 95):.0f} ms, "
      f"p99 {np.percentile(t, 99):.0f} ms")

merged = [b for b in budget if not b[0].startswith("feature lookup")]
merged.append(("one batched feature lookup", 14, 95))
t2 = simulate(merged)
print(f"after merging the lookups: p50 {np.percentile(t2, 50):.0f} ms, "
      f"p95 {np.percentile(t2, 95):.0f} ms, p99 {np.percentile(t2, 99):.0f} ms")

end to end: p50 63 ms, p95 134 ms, p99 200 ms
after merging the lookups: p50 42 ms, p95 83 ms, p99 124 ms

So the honest end-to-end p99 is 200 ms against a 308 ms sum, and the service is over its budget by a third rather than by a factor of two. Merging the two feature lookups into one batched call — one round trip, one connection, the aggregate precomputed into the same store — takes the p99 to 124 ms and lands the service inside its target. No model work, no new hardware, one query.

Three habits come out of this and they transfer to any serving path. Write the budget before optimising, because the component you would have guessed is almost never the one. Never sum tail percentiles and call it a latency; either simulate or measure end to end. And count round trips before counting milliseconds, because on a path like this the number of sequential network hops predicts the tail better than any single component's speed.