Timeouts, retries and exactly-once, in practice

Every call from one service to another can be slow, can fail, or can succeed in a way the caller never hears about. This section is the standard set of defences: a timeout on every call, retries only where repeating the work is safe, idempotency keys to make it safe, and a circuit breaker so a struggling dependency is not hammered while it tries to recover. It also settles the exactly-once question, which is that you cannot have it over a network, so real systems deliver at least once and make the handler tolerate a duplicate.

Timeouts#

A call without a timeout can wait forever, holding a thread, a connection and a user. Set timeouts on every network call, and make them fit inside the caller's own deadline, so a request with a 10 second budget does not make three sequential calls with 10 second timeouts each.

Retries#

• Retry on timeouts, connection errors and explicit 'try again' responses such as 429 and 503.
• Do not retry on 400-type errors; the request is wrong and will stay wrong.
• Use exponential backoff with jitter, and a limit.
• Only retry operations that are safe to repeat, which usually means making them idempotent.

There is a fifth rule that is missing from most lists and causes most retry-related outages: cap the total retry traffic, not just the retries per request. Retries are demand, and they are demand that appears exactly when the system has least capacity to serve it. A service that fails under a brief spike, whose clients then retry, faces a load made of the original traffic plus everything that failed, which fails too, and retries again.

import collections

CAPACITY = 100                 # requests a second the server can actually complete

def simulate(retries, budget, seconds=600):
    pending = collections.defaultdict(int)
    trace = []
    for t in range(seconds):
        base = 140 if 60 <= t < 120 else 80     # a one-minute spike, then it is over
        offered = base + pending.pop(t, 0)
        trace.append(offered)
        failed = max(0, offered - CAPACITY)
        if retries:
            pending[t + 2] += min(failed, int(CAPACITY * budget))
    return trace

print("the spike itself lasts 60 s and is 1.4x capacity")
for label, retries, budget in [("no retries", 0, 0.0),
                               ("retry, no budget", 1, 10.0),
                               ("retry budget 10% of capacity", 1, 0.10)]:
    trace = simulate(retries, budget)
    recovered = next((t for t in range(120, 600) if trace[t] <= CAPACITY), None)
    print(f"{label:30} peak offered {max(trace):5}/s   "
          f"still overloaded at t=150: {trace[150]:4}/s   "
          f"recovered at t={recovered}")

# the spike itself lasts 60 s and is 1.4x capacity
# no retries                     peak offered   140/s   still overloaded at t=150:   80/s   recovered at t=120
# retry, no budget               peak offered  1140/s   still overloaded at t=150:  780/s   recovered at t=218
# retry budget 10% of capacity   peak offered   150/s   still overloaded at t=150:   80/s   recovered at t=120

The fix is a retry budget: the client tracks its own ratio of retries to first attempts and stops retrying once it passes some small fraction, typically around ten per cent. The third row is that, and it barely registers the spike. The related habit is to make retries decorrelated — exponential backoff with full jitter, meaning a random wait between zero and the current backoff rather than the backoff itself, so a thousand clients that failed together do not come back together.

Idempotency keys#

The client generates a unique key per logical operation and sends it with every attempt. The server records the key with the result, and a repeated key returns the stored result instead of acting again. This is how payment APIs make 'charge the card' safe to retry.

The word doing the work in that paragraph is 'per logical operation', and it is where the implementation goes wrong. The key must be generated once, outside the retry loop, and reused by every attempt at the same payment. Generate it inside the loop and you have a unique key per attempt, which is the same as having no key at all — and the code looks correct, because it does use idempotency keys.

import random, uuid

class Ledger:
    def __init__(self):
        self.charges, self.by_key = [], {}

    def charge(self, account, pence, key):
        if key in self.by_key:                 # replay, not a new instruction
            return self.by_key[key], "replayed"
        charge_id = len(self.charges)
        self.charges.append((account, pence))  # must be one transaction with the line below
        self.by_key[key] = charge_id
        return charge_id, "charged"

def flaky_call(ledger, account, pence, key):
    result = ledger.charge(account, pence, key)
    if random.random() < 0.4:
        raise TimeoutError("reply lost, the charge may or may not have happened")
    return result

random.seed(9)
for label, fresh_key_each_try in [("new key on every retry", True),
                                  ("one key per payment", False)]:
    ledger = Ledger()
    for payment in range(100):
        key = str(uuid.uuid5(uuid.NAMESPACE_OID, f"payment-{payment}"))
        for attempt in range(4):
            try:
                if fresh_key_each_try:
                    key = str(uuid.uuid4())
                flaky_call(ledger, "acct-1", 999, key)
                break
            except TimeoutError:
                continue
    total = sum(p for _, p in ledger.charges)
    print(f"{label:24} 100 payments of 999p -> charged {total:7,}p "
          f"in {len(ledger.charges):3} rows  "
          f"({'OVERCHARGED' if total > 99900 else 'correct'})")

# new key on every retry   100 payments of 999p -> charged 172,827p in 173 rows  (OVERCHARGED)
# one key per payment      100 payments of 999p -> charged  99,900p in 100 rows  (correct)

Circuit breakers#

When a dependency is failing, retrying hard makes it worse. A circuit breaker counts failures and, past a threshold, fails fast for a cool-down period without calling the dependency, then lets a few test requests through. Your service degrades instead of hanging.

It has three states and the third is the one people forget. Closed means calls go through and failures are counted. Open means calls fail immediately without touching the dependency. Half-open, after the cool-down, means one probe is allowed: if it succeeds the breaker closes, and if it fails the breaker reopens at once rather than waiting for another five failures.

TIMEOUT = 2.0        # seconds a call to a dead dependency burns before giving up

class CircuitBreaker:
    def __init__(self, threshold=5, cooldown=10.0):
        self.threshold, self.cooldown = threshold, cooldown
        self.failures, self.opened_at = 0, None

    def state(self, now):
        if self.opened_at is None:
            return "closed"
        return "half-open" if now - self.opened_at >= self.cooldown else "open"

    def call(self, now, dependency_alive):
        state = self.state(now)
        if state == "open":
            return 0.0, "fast fail"                  # no thread held, no wait
        if dependency_alive:
            self.failures, self.opened_at = 0, None  # a probe that works closes it
            return 0.01, "ok"
        if state == "half-open":
            self.opened_at = now                     # one bad probe reopens it at once
            return TIMEOUT, "timed out"
        self.failures += 1
        if self.failures >= self.threshold:
            self.failures, self.opened_at = 0, now
        return TIMEOUT, "timed out"

def run(threshold):
    cb = CircuitBreaker(threshold=threshold)
    waited, reached = 0.0, 0
    for tick in range(600):                          # 60 s at 10 calls a second
        now = tick / 10
        alive = not (10.0 <= now < 40.0)             # the dependency is down for 30 s
        cost, what = cb.call(now, alive)
        waited += cost
        if what == "timed out":
            reached += 1
    return waited, reached

for label, threshold in [("no breaker", 10 ** 9),
                         ("breaker: 5 failures, 10 s open", 5)]:
    waited, reached = run(threshold)
    print(f"{label:32} thread-seconds waiting {waited:7.1f}   "
          f"calls that hit the dead dependency {reached:3} of 300")

# no breaker                       thread-seconds waiting   603.0   calls that hit the dead dependency 300 of 300
# breaker: 5 failures, 10 s open   thread-seconds waiting    17.0   calls that hit the dead dependency   7 of 300

Two things to say when this comes up. A breaker is only useful if there is something sensible to do when it is open — serve stale data, return a partial response, queue the work for later — because failing fast is still failing, and a breaker in front of a dependency you cannot do without just converts a slow outage into a quick one. And the threshold should be a failure rate over a window rather than a raw count, or a low-traffic endpoint trips on five failures spread across an hour.

Exactly once is a myth; idempotent is not#

Over a network you cannot guarantee a message is delivered exactly once: an acknowledgement can be lost after the work was done. Real systems deliver at least once and make handlers idempotent, so a duplicate does no harm.

The outbox pattern#

You need to save an order and publish an 'order created' event. Doing both separately means one can succeed and the other fail. Instead, write the event to an outbox table in the same database transaction as the order; a separate process publishes outbox rows and marks them sent. The event may be published twice, which idempotent consumers handle.

Trace the crash to see why it works. Crash before the transaction commits and neither the order nor the event exists, which is consistent. Crash after the commit and both exist, with the event unsent, and the publisher picks it up on its next pass. Crash after publishing but before marking the row sent, and the event is published a second time — which is the case you deliberately chose, because a duplicate delivered to an idempotent consumer costs nothing and a lost event costs an order nobody ever hears about. That is the whole design principle of this section: you cannot remove uncertainty, so you arrange for the surviving failure to be the harmless one.

What to keep and what to skip#

• Keep: a timeout on every call, sized to fit inside the caller's own deadline.
• Keep: retries are demand, so cap the total retry rate and jitter the backoff. Unbudgeted retries turn a spike into an outage that outlives its cause.
• Keep: one idempotency key per logical operation, generated outside the retry loop, and stored atomically with the effect.
• Keep: exactly-once delivery does not exist over a network. At-least-once plus an idempotent handler is the answer.
• Keep: the outbox pattern and the three crash points that make it correct.
• Skip: tuning circuit breaker constants in the abstract. Five failures and thirty seconds is a fine starting point; the numbers come from your dependency's recovery time.
• Skip: arguing about which broker gives exactly-once semantics. Say where the boundary of that claim is and move on.