Retries, rate limits and timeouts

Every call to a model provider will fail sooner or later, and most of those failures are temporary: a rate limit, an overloaded backend, a network blip. Handling them is not optional engineering polish, because a service that gives up on the first 429 is a service that is down whenever the provider is busy. This section covers retrying in a way that helps rather than hurts, reading the rate limit headers the provider already sends you, setting timeouts at every level, and stopping entirely when the thing you are calling is genuinely broken.

What is worth retrying#

status   meaning                     retry?
408      request timeout             yes
429      rate limited                yes, and read Retry-After
500      server error                yes
502/503  bad gateway / unavailable   yes
504      gateway timeout             yes
400      malformed request           no. it will fail identically
401/403  auth                        no. fix the credential
404      not found                   no
422      semantically invalid        no

Exponential backoff with jitter#

Backoff means waiting longer after each failure — one second, two, four, eight — so that a struggling service gets progressively more room. Jitter means randomising that wait. The second half is the one people leave out, and it is the half that matters when more than one client is involved, because without it every client that failed at the same moment retries at the same moment, forever.

import random

def simulate(jitter, clients=100, capacity=20, seed=0):
    """Every client fails at t=0 and retries. capacity is how many the server
    can serve at one instant. Returns the size of each arrival burst."""
    rng = random.Random(seed)
    pending = [(0.0, 0) for _ in range(clients)]     # (next attempt, attempt no.)
    bursts = []
    while pending:
        t = min(p[0] for p in pending)
        due = [p for p in pending if p[0] <= t + 1e-9]
        bursts.append(len(due))
        rest = due[min(len(due), capacity):]         # the rest are rejected again
        pending = [p for p in pending if p[0] > t + 1e-9]
        for _, n in rest:
            base = 2 ** n                            # 1, 2, 4, 8 seconds
            pending.append((t + (rng.uniform(0, base) if jitter else base), n + 1))
        if len(bursts) > 200:
            break
    return bursts, t

for jitter in (False, True):
    bursts, done = simulate(jitter)
    print(jitter, bursts[:6], "cleared at t =", round(done, 2))

# False [100, 80, 60, 40, 20]      cleared at t = 15.0
# True  [100, 1, 1, 1, 1, 1]       cleared at t = 0.98

That synchronised wave is the thundering herd, and the reason it matters more than it looks is that the clients caused the problem they are waiting for. Full jitter — a uniform random wait between zero and the backoff ceiling, which is what the code above does — is the simplest version and is hard to improve on. The production form of this adds two things: a cap on the delay so it does not grow to minutes, and a limit on attempts so a request cannot retry forever.

import time

RETRYABLE = {408, 429, 500, 502, 503, 504}

def retryable(err):
    """Could the identical request succeed if sent again?"""
    return getattr(err, "status_code", None) in RETRYABLE

def with_retries(call, attempts=5, cap=30.0):
    for n in range(attempts):
        try:
            return call()
        except Exception as e:
            if n == attempts - 1 or not retryable(e):
                raise
            delay = min(cap, 2 ** n)
            time.sleep(random.uniform(0, delay))     # full jitter
    raise RuntimeError("unreachable")

Respecting rate limit headers#

Guessing the backoff is the fallback. Most providers tell you the answer directly, and a client that reads the headers is better behaved and faster than one that does not — it waits exactly as long as required rather than doubling blindly past it.

Retry-After                  seconds, or a date. Obey it exactly.
x-ratelimit-limit-requests   your ceiling for this window
x-ratelimit-remaining-requests  how many you have left
x-ratelimit-reset-requests   when the window refills
x-ratelimit-remaining-tokens most providers limit tokens as well as calls

The remaining counters are the more useful half, because they let you avoid the error rather than recover from it. Read them on every response and slow down as the remaining count approaches zero, and a well-behaved client rarely sees a 429 at all. The token counter is the one that surprises people: you can be far inside the request limit and still be throttled because a few long prompts consumed the token budget.

Timeouts at every level#

A request with no timeout can hang until the operating system gives up, which may be minutes, and during that time it holds a connection, a worker and a user's attention. The mistake is setting one timeout and believing you are covered, when there are several and they nest.

connect timeout    establishing the TCP connection     ~5 s
read timeout       waiting for the next byte           model calls: 60 s+
total timeout      the whole operation                 bounds everything
retry budget       total time across all attempts      what the caller feels
request deadline   propagated from the incoming call   the honest ceiling

Circuit breaking#

Retrying assumes the failure is temporary. When it is not — the provider is down, the credential has been revoked, the region is unreachable — retrying makes everything worse: each request waits through its full backoff before failing, workers pile up, and a dependency that is merely broken becomes an outage in your own service. A circuit breaker notices a run of failures and stops calling for a while.

closed      calls pass through            counting failures
open        calls fail immediately       no waiting, no load on the dependency
half-open   one trial call is allowed    success closes it, failure re-opens

Open it on a failure rate over a window rather than on a count of consecutive failures, or a steady trickle of successes will hold it closed while most requests fail. And make what happens in the open state a deliberate decision rather than an exception — that is exactly where the fallback chain from the model-selection topic belongs.