Designing so failure is survivable
By the end: Your agent degrades rather than collapses, and mistakes are recoverable.
- Idempotent tools
- Dry run and approval gates
- Checkpointing and resuming
- Verification as a step
New to agent reliability? Agents fail in ways ordinary programs do not. A normal bug raises an error; an agent calls the same broken tool forty times, or acts confidently on something it got wrong, or stops halfway and reports success. Agent reliability is the set of defences against that: budgets on turns and spend, tools that are safe to run twice, dry runs, saved checkpoints so a failed run can resume, and a person asked to approve anything destructive. Why it matters
You can name the failures; this is how you make them survivable. The aim is not an agent that never goes wrong, because that is not on offer. It is an agent whose mistakes can be retried without doing the thing twice, stopped before anything irreversible, resumed from where a long run died, and checked before it claims to be finished. Each is a small piece of engineering, and their absence is what turns a bad run into an incident with a manual cleanup at the end of it.
Idempotency#
A retried tool call must not do the thing twice. This matters more with agents than with ordinary code, because retries happen for reasons you did not plan: a timeout where the action actually succeeded, a resumed run, a model repeating itself.
def issue_refund(order_id: str, idempotency_key: str):
if already_processed(idempotency_key):
return previous_result(idempotency_key) # safe to call again
result = payments.refund(order_id)
record(idempotency_key, result)
return resultThe cost of getting this wrong is not abstract, so here it is with the money counted. The payment provider below always applies the charge and sometimes fails to return a response, which is the ordinary behaviour of a network. The naive version treats the timeout as a failure and tries again.
LEDGER = [] # every charge the payment system actually made
PROCESSED = {} # idempotency key -> the result we returned
def payments_refund(order_id, cents, flaky):
"""The charge lands. Whether the response comes back is another matter."""
LEDGER.append((order_id, cents))
if flaky:
raise TimeoutError("no response from the payment provider")
return {"refund_id": "R-" + str(len(LEDGER)), "cents": cents}
def refund_naive(order_id, cents, flaky):
try:
return payments_refund(order_id, cents, flaky)
except TimeoutError:
return payments_refund(order_id, cents, False) # "it failed, try again"
def refund_idempotent(order_id, cents, key, flaky):
if key in PROCESSED:
return PROCESSED[key]
try:
result = payments_refund(order_id, cents, flaky)
except TimeoutError:
# Unknown outcome. Ask the payment system what happened, do not retry.
prior = [c for o, c in LEDGER if o == order_id]
result = {"refund_id": "recovered", "cents": prior[-1]} if prior else None
if result is None:
raise
PROCESSED[key] = result
return result
for label, fn in (("naive retry", lambda: refund_naive("A-1", 2900, flaky=True)),
("idempotent", lambda: refund_idempotent("A-2", 2900,
key="run-77:refund:A-2",
flaky=True))):
LEDGER.clear(); PROCESSED.clear()
fn()
charged = sum(c for _, c in LEDGER)
print(f"{label:<14}{len(LEDGER)} charges {charged} cents {LEDGER}")
# the same call arriving twice, which is what a resumed run looks like
LEDGER.clear(); PROCESSED.clear()
for _ in range(3):
refund_idempotent("A-3", 2900, key="run-77:refund:A-3", flaky=False)
print(f"{'replayed x3':<14}{len(LEDGER)} charges {sum(c for _, c in LEDGER)} cents")
# naive retry 2 charges 5800 cents [('A-1', 2900), ('A-1', 2900)]
# idempotent 1 charges 2900 cents [('A-2', 2900)]
# replayed x3 1 charges 2900 centsTwo details make this work in practice. The key has to be derived, not generated: something like the run id, the tool name and a hash of the arguments, so that the same intent in the same run produces the same key while a second genuine refund on the same order produces a different one. Generating a fresh uuid per call is the common mistake and defeats the whole mechanism, because the retry then carries a key nobody has seen.
And the recovery path matters more than the cache. A timeout is an unknown outcome, so the right move is to ask the downstream system what it has, by idempotency key if it supports them and by a search for a matching recent record if it does not. Retrying blind is only safe when the operation is naturally idempotent, which is to say when it sets a value rather than changing one: setting a status to refunded is safe to repeat, adding 29.00 to a balance is not.
Dry run#
Let the agent produce a plan of actions without executing them. Useful in development, useful in testing, and useful in production when a person should see what is about to happen.
DRY_RUN = True
def refund_tool(order_id, amount, idempotency_key):
if DRY_RUN:
return {"ok": True, "would_have": f"refunded {order_id} for {amount}"}
return refund_idempotent(order_id, amount, idempotency_key, flaky=False)
print(refund_tool("A-4471", 2900, "run-77:refund:A-4471"))
# {'ok': True, 'would_have': 'refunded A-4471 for 2900'}Approval gates#
Anything irreversible or externally visible should pause for a person. The agent proposes, a human confirms, then it executes. Choose the threshold deliberately: everything needing approval defeats the purpose, and nothing needing it is how money moves without anyone deciding.
- Always gate: sending money, deleting data, anything that reaches a customer.
- Gate above a threshold: refunds over an amount, bulk operations over a count.
- Do not gate: reads, searches, drafts, anything reversible in one click.
That threshold is usually argued about on instinct, and it does not have to be. Write down the traffic, write down what a wrong action costs to clean up, and the three policies separate immediately. The harm figures below are the part to argue about; the arithmetic on top of them is not.
# One thousand actions from a week of traffic, with what each one costs to
# clean up if the agent gets it wrong. The harm figures are yours to argue
# about; the arithmetic is not.
ACTIONS = [
# kind, count, external, cents, bulk, harm_if_wrong
("search_orders", 620, False, 0, 0, 0),
("get_customer", 180, False, 0, 0, 0),
("draft_reply", 60, False, 0, 0, 0),
("send_email", 70, True, 0, 0, 40),
("refund_small", 45, False, 900, 0, 9),
("refund_large", 8, False, 24000, 0, 240),
("bulk_close", 14, False, 0, 120, 600),
("delete_account", 3, False, 0, 0, 2000),
]
P_WRONG = 0.03 # how often the agent picks the wrong target
def gate_none(a): return False
def gate_all(a): return a[5] >= 0
def gate_rule(a):
kind, _, external, cents, bulk, _ = a
if kind == "delete_account": return True # always: irreversible
if external: return True # always: reaches a customer
if cents > 5000: return True # threshold: large amounts
if bulk > 50: return True # threshold: bulk operations
return False
print(f"{'policy':<14}{'approvals':>11}{'harm left':>12}{'harm stopped':>14}"
f"{'per approval':>14}")
baseline = sum(n * P_WRONG * harm for _, n, _, _, _, harm in ACTIONS)
for name, policy in (("gate nothing", gate_none), ("gate everything", gate_all),
("by rule", gate_rule)):
approvals = sum(n for a in ACTIONS for n in [a[1]] if policy(a))
left = sum(n * P_WRONG * harm for a in ACTIONS
for _, n, _, _, _, harm in [a] if not policy(a))
stopped = baseline - left
per = stopped / approvals if approvals else float("nan")
print(f"{name:<14}{approvals:>11}{left:>11.0f} {stopped:>13.0f}{per:>14.2f}")
print()
print("gated by rule:", [a[0] for a in ACTIONS if gate_rule(a)])
# policy approvals harm left harm stopped per approval
# gate nothing 0 586 0 nan
# gate everything 1000 0 586 0.59
# by rule 95 12 574 6.04
#
# gated by rule: ['send_email', 'refund_large', 'bulk_close', 'delete_account']The last column is the one to take to the argument, because gating everything is not the safe option, it is the option where approvals stop being read. A reviewer asked to approve a thousand searches approves the thousand-and-first thing without looking, and the delete arrives in that queue. Reserving the interruption for the ninety-five actions that carry the harm is what keeps the review real.
In code, the gate belongs in the dispatcher rather than inside each tool, so that a new tool is gated by default rather than by remembering. The shape that works is a policy function returning allow, deny or needs-approval; the loop pauses on needs-approval and persists the pending action, because a human is not going to answer within the request. That means an approval gate implies durable state, which is the same state a resumable run needs, and the two are usually built together.
Checkpoint and resume#
Save state after each turn so a crash loses one turn rather than the whole run. The subtlety is that the message list is not the whole state: if a tool changed something in the world, replaying that turn repeats the change, which is where idempotency earns its place.
CHECKPOINTS = {}
def step(messages):
return {"turn": len(messages)} # stands in for the model call
def apply(response, messages):
return messages + [response]
def save_checkpoint(run_id, turn, messages):
CHECKPOINTS[run_id] = (turn, list(messages))
run_id, messages, max_turns = "run-77", [], 4
for turn in range(max_turns):
response = step(messages)
messages = apply(response, messages)
save_checkpoint(run_id, turn, messages) # after, not before
print(CHECKPOINTS["run-77"][0], "turns saved,", len(CHECKPOINTS["run-77"][1]), "messages")
# 3 turns saved, 4 messagesSaving after the turn rather than before is the right choice and it is not sufficient, which is worth seeing rather than being told. There is a window between the side effect landing and the checkpoint being written, and no ordering closes it: whatever you write first, the process can die in between. The run below is a five-step plan killed at step three, resumed from whatever the checkpoint said.
PLAN = ["refund A-9", "email ada", "close T-9", "note in CRM", "update forecast"]
class Crash(Exception):
pass
def attempt(start, effects, seen, idempotent, save, crash_at=None, in_window=False):
"""Runs the plan from 'start'. A crash in the window happens after the side
effect and before the checkpoint is written, which is the gap no ordering
can close."""
saved = start
for i in range(start, len(PLAN)):
if save == "before":
saved = i
key = "run-77:" + PLAN[i]
if not (idempotent and key in seen):
effects.append(PLAN[i])
seen.add(key)
if crash_at == i and in_window:
raise Crash(saved)
if save == "after":
saved = i + 1
if crash_at == i:
raise Crash(saved)
return saved
def run(save, idempotent, in_window):
effects, seen = [], set()
try:
attempt(0, effects, seen, idempotent, save, crash_at=2, in_window=in_window)
except Crash as c:
attempt(c.args[0], effects, seen, idempotent, save)
dupes = sorted({s for s in effects if effects.count(s) > 1})
return f"{len(effects)} effects {dupes}"
print(f"{'policy':<30}{'crash between steps':>24}{'crash in the window':>24}")
for label, save, idem in (("checkpoint before the step", "before", False),
("checkpoint after the step", "after", False),
("after, and idempotent", "after", True)):
print(f"{label:<30}{run(save, idem, False):>24}{run(save, idem, True):>24}")
print("five steps, so five effects is correct")
# policy crash between steps crash in the window
# checkpoint before the step 6 effects ['close T-9'] 6 effects ['close T-9']
# checkpoint after the step 5 effects [] 6 effects ['close T-9']
# after, and idempotent 5 effects [] 5 effects []
# five steps, so five effects is correctOnly the last row is correct in both columns, and it is correct because of the previous section rather than this one. Checkpointing decides how much work a crash costs you; idempotency decides whether the recovery is safe. Neither substitutes for the other, and the pair is what makes a long run resumable at all.
What to put in the checkpoint is the other half. The message list is the obvious part and the least interesting: also save the run id, the budget consumed so far, the idempotency keys already used with their results, any pending approval, and a version string for the prompt and tool definitions. That last one stops a run resuming into a different agent after a deploy, which produces a transcript that makes no sense to anybody reading it afterwards.
Detecting a stuck loop, and what each detector misses#
The agent-loop topic puts a repeated-call detector in the loop and shows what it saves. The question left open there is how much of the problem it actually catches, because a stuck agent does not always repeat itself exactly. Four kinds of stuck run and four detectors, over four hundred runs of each kind, answer it. The healthy runs include a legitimate retry about a quarter of the time, which is the realistic thing that makes this hard.
import numpy as np
N_TURNS = 16
def make_run(kind, rng):
"""Returns a list of (tool, args, gained_a_new_fact) for one run."""
start = int(rng.integers(2, 6))
turns = [("get_order", {"id": "A-" + str(t)}, True) for t in range(start)]
for t in range(start, N_TURNS):
i = t - start
if kind == "exact":
turns.append(("search", {"q": "refund policy"}, False))
elif kind == "reworded":
q = ["refund policy", "Refund Policy", "policy refund",
"REFUND POLICY"][i % 4]
turns.append(("search", {"q": q}, False))
elif kind == "cycle":
turns.append(("search", {"q": "refund"}, False) if i % 2 == 0
else ("read_doc", {"id": "POL-14"}, False))
elif kind == "drift":
turns.append(("search", {"q": "refund question " + str(i)}, False))
else: # healthy, with a legitimate retry
retry = rng.random() < 0.25
turns.append(("get_order", {"id": "A-" + str(t - retry)}, not retry))
return turns
def canon(args):
return tuple(sorted(" ".join(sorted(str(v).lower().split())) for v in args.values()))
def detect(turns, mode, window=6, patience=4):
"""First turn at which this detector fires, or None."""
seen, last, repeats, since_progress = [], None, 0, 0
for t, (name, args, gained) in enumerate(turns):
sig = (name, tuple(sorted(args.items()))) if mode in ("exact", "window") \
else (name, canon(args))
if mode in ("exact", "canonical"):
repeats = repeats + 1 if sig == last else 0
last = sig
if repeats >= 2:
return t
elif mode == "window":
if sig in seen[-window:]:
return t
seen.append(sig)
else: # no progress
since_progress = 0 if gained else since_progress + 1
if since_progress >= patience:
return t
return None
rng = np.random.default_rng(0)
KINDS = ("exact", "reworded", "cycle", "drift", "healthy")
MODES = ("exact", "canonical", "window", "progress")
RUNS = {k: [make_run(k, rng) for _ in range(400)] for k in KINDS}
print(f"{'detector':<12}" + "".join(f"{k:>11}" for k in KINDS))
for mode in MODES:
cells = []
for kind in KINDS:
hits = [detect(r, mode) for r in RUNS[kind]]
rate = sum(h is not None for h in hits) / len(hits)
cells.append(f"{rate:>10.0%} ")
print(f"{mode:<12}" + "".join(cells))
# detector exact reworded cycle drift healthy
# exact 100% 0% 0% 0% 0%
# canonical 100% 100% 0% 0% 0%
# window 100% 100% 100% 0% 98%
# progress 100% 100% 100% 100% 2%Read the table as a progression and it stops being obvious. Exact argument equality catches one stuck pattern in four. Canonicalising the arguments, twenty minutes of work, doubles that. A window catches cycles as well, and is unusable: an agent that reads the same order twice in six turns is not stuck, it is working, and a detector that stops those runs is worse than no detector. Only the progress check catches the drifting agent, the one issuing a genuinely different search every turn and learning nothing, which is the hardest kind to see in a transcript and the most common in practice.
So define progress explicitly instead of inferring it from the calls. Something is progress if it puts a new fact in the run's ledger: a record retrieved that was not there before, a side effect applied, an open question answered. Count the turns since the last one, and when the count passes a small number, intervene. The detector is a handful of lines, it does not care how the model phrases things, and it is the only one on that table you can leave switched on.
print("median turn the detector fires on a stuck run (16-turn runs):")
for mode in MODES:
fired = [detect(r, mode) for kind in ("exact", "reworded", "cycle", "drift")
for r in RUNS[kind]]
fired = [f for f in fired if f is not None]
print(f" {mode:<10}{np.median(fired):>5.1f} fired on {len(fired)} of 1600 stuck runs")
# exact 6.0 fired on 400 of 1600 stuck runs
# canonical 6.0 fired on 800 of 1600 stuck runs
# window 6.0 fired on 1200 of 1600 stuck runs
# progress 7.0 fired on 1600 of 1600 stuck runsVerify before reporting success#
Add an explicit verification step that checks the world rather than asking the model whether it finished. It is cheap, it catches partial completion, and it converts a silent failure into a clear one.
from dataclasses import dataclass, field
@dataclass
class Requirement:
name: str
check: callable
@dataclass
class Task:
requirements: list = field(default_factory=list)
@dataclass
class AgentResult:
complete: bool
missing: list
STATE = {"refunded": True, "emailed": True, "crm_note": False}
task = Task([Requirement("order refunded", lambda: STATE["refunded"]),
Requirement("customer emailed", lambda: STATE["emailed"]),
Requirement("reason in CRM", lambda: STATE["crm_note"])])
def verify(task) -> tuple[bool, list[str]]:
missing = [r.name for r in task.requirements if not r.check()]
return (not missing), missing
ok, missing = verify(task)
print(AgentResult(complete=ok, missing=missing) if not ok else "verified")
# AgentResult(complete=False, missing=['reason in CRM'])Never let the model grade its own completion
Asking the agent whether it finished uses the same process that produced the mistake, and it will say yes. Verification has to be code checking real state.
Where people go wrong
Checkpointing before executing the turn rather than after. On resume the turn runs a second time, and without idempotency the side effect happens twice.
If this comes up in an interview
How do you make an agent safe to retry?
Idempotent tools keyed on run id and intent, so a repeated call returns the previous result instead of acting twice. Checkpoint after each turn so a resumed run continues rather than restarting, and remember that the message list is not the whole state when a tool changed the world.
Then they push further
The payment API does not support idempotency keys. What do you do?
Record intent before calling: write a pending record with a unique id and check for it before any retry. Resolve uncertain cases by asking the payment system what actually happened instead of calling again.
What gets marked up. Strong answers mention idempotency keys and checking state before retrying, and treat a timeout as an unknown outcome rather than a failure.
Which agent actions need human approval?
Anything irreversible or externally visible: money, deletion, anything reaching a customer. Threshold-gate the rest by amount or volume. Gating everything defeats the purpose and gating nothing is how unreviewed actions happen.
How do you confirm an agent actually completed a task?
Verify against the world in code, not by asking the model. Check that each required side effect happened. Asking the agent uses the same process that produced any mistake, so it will report success either way.
Build this: Add idempotency keys, checkpointing and a verification step to an agent. Then kill the process mid-run, resume it, and confirm nothing happened twice and the verification still passes.
Watch it explained
This is one section of Agent reliability, 8 hours across 2 sections. Read it in full to get the whole topic in order.
Stop 20 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.