Agent reliability
Degrading instead of collapsing.
Agents fail in ways ordinary programs do not. A normal bug raises an error; an agent calls the same broken tool forty times, or acts confidently on something it got wrong, or stops halfway and reports success. Agent reliability is the set of defences against that: budgets on turns and spend, tools that are safe to run twice, dry runs, saved checkpoints so a failed run can resume, and a person asked to approve anything destructive.
The failure that costs you is not the crash, it is the one that looks like a success — the refund issued twice, the half-finished migration reported as done. And because the model's choices vary between runs, the same task can fail on Tuesday in a way it did not on Monday, so a run that worked once is not evidence that it works.
It sits on the agent loop and is most of the distance between a demo and something people depend on. The tools it hardens come from tool calling and MCP; the budgets it enforces are the same ones context management is trying to stay inside. Observability is how you see what a run did, and evaluation is how you find out whether it did it well.
Design for the failure that looks like success, not the one that raises an error.
In more detailAgents fail in ways ordinary software does not: infinite tool loops, confidently wrong actions, and partial completion that looks like success. Defences are idempotent tools, dry-run modes, explicit budgets, checkpointing so a failed run can resume, and human approval gates on anything destructive.
Learn first
How agents fail
By the end: You can name the ways an agent goes wrong and recognise each one in a trace.
- Failures that raise, and failures that do not
- Partial completion
- The confident wrong action
- Compounding error over turns
Ordinary software mostly fails loudly: something raises, something times out, and you get an alert. Agents do that too, and those are the easy cases. The failures that cost you are the ones where nothing errors at all — the run completes, the agent reports success, and the work is half done or confidently wrong. This section gives you a vocabulary for those failures and shows how each one looks in a trace, which is what you need before you can design against them.
Two categories#
Some agent failures look like ordinary software failures: a timeout, a rate limit, a tool raising. Those are the easy ones, because something tells you.
The dangerous category is the one where nothing errors. The agent finishes, reports success, and the work is wrong. Nothing in your logs says so.
There is a third category between them that is worth separating out, because people keep putting it in the first one. A tool call that times out has not failed. It has an unknown outcome: the request may have arrived, been processed and charged, and only the response gone missing. Treating a timeout as a failure and retrying is how a customer gets refunded twice, and it is the most common way money moves incorrectly in an agent system. The next section is largely about that.
Partial completion#
An agent asked to do four things does three, then summarises as though it did four. This is common and it is easy to miss, because the summary is confident and reads well.
Task: "Refund the order, email the customer, close the ticket,
and note the reason in the CRM."
Trace: refund_order ok
send_email ok
close_ticket ok
(no CRM call)
Answer: "Done. The customer has been refunded and notified,
the ticket is closed and the reason is recorded."The defence is not to trust the summary. Check the world: did the record actually change? That is also exactly what an agent eval should assert.
Worth running once, because the shape of it is more convincing than the description. Below, every tool returns ok and one of them writes nothing, which is what a stubbed integration, a silently-swallowed exception or a permissions problem all look like from inside the loop.
WORLD = {"orders": {"A-1": "shipped"}, "emails": [], "tickets": {"T-1": "open"},
"crm": {}}
def refund(order_id): WORLD["orders"][order_id] = "refunded"; return {"ok": True}
def send_email(to): WORLD["emails"].append(to); return {"ok": True}
def close_ticket(tid): WORLD["tickets"][tid] = "closed"; return {"ok": True}
def note_in_crm(cid, text):
return {"ok": True} # the bug: reports success, writes nothing
TOOLS = {"refund": refund, "send_email": send_email,
"close_ticket": close_ticket, "note_in_crm": note_in_crm}
CALLS = [("refund", ("A-1",)), ("send_email", ("ada@example.com",)),
("close_ticket", ("T-1",)), ("note_in_crm", ("C-118", "refunded, faulty"))]
results = [(name, TOOLS[name](*args)) for name, args in CALLS]
for name, result in results:
print(f"{name:<14}{result}")
SUMMARY = ("Done. The customer has been refunded and notified, the ticket is "
"closed and the reason is recorded in the CRM.")
REQUIREMENTS = {
"order refunded": lambda: WORLD["orders"]["A-1"] == "refunded",
"customer emailed": lambda: WORLD["emails"] == ["ada@example.com"],
"ticket closed": lambda: WORLD["tickets"]["T-1"] == "closed",
"reason in CRM": lambda: "C-118" in WORLD["crm"],
}
missing = [name for name, check in REQUIREMENTS.items() if not check()]
print()
print("the agent says:", SUMMARY)
print("every tool call returned ok:", all(r["ok"] for _, r in results))
print("requirements actually met :", len(REQUIREMENTS) - len(missing), "of",
len(REQUIREMENTS), " missing:", missing)
# refund {'ok': True}
# send_email {'ok': True}
# close_ticket {'ok': True}
# note_in_crm {'ok': True}
#
# the agent says: Done. The customer has been refunded and notified, the ticket is closed and the reason is recorded in the CRM.
# every tool call returned ok: True
# requirements actually met : 3 of 4 missing: ['reason in CRM']Write the requirements down as a dict of checks when you write the task, not afterwards. It costs a few minutes, it gives the agent eval its assertions for free, and it converts the most expensive failure mode in this section into a returned error.
The confident wrong action#
The agent refunds the wrong order, emails the wrong customer, or updates the wrong record. Everything succeeded technically. The damage is real and reversing it is a support problem rather than a code problem.
Compounding error#
Each turn builds on the previous ones. A small mistake at turn two becomes an assumption at turn three and a conclusion at turn six. By the end the agent is confidently working on a problem that does not exist.
This is arithmetic rather than folklore, and the arithmetic is worth knowing because it decides how long a run you can allow. If each turn has an independent chance of introducing something wrong, and a wrong fact poisons everything conditioned on it, the chance of a clean run is that per-turn success rate raised to the number of turns.
print("chance the whole run is clean, given a per-turn error rate")
print(f"{'turns':>6}" + "".join(f"{p:>10.0%}" for p in (0.01, 0.02, 0.05, 0.10)))
for n in (3, 5, 10, 20, 40):
print(f"{n:>6}" + "".join(f"{(1 - p) ** n:>10.1%}" for p in (0.01, 0.02, 0.05, 0.10)))
# chance the whole run is clean, given a per-turn error rate
# turns 1% 2% 5% 10%
# 3 97.0% 94.1% 85.7% 72.9%
# 5 95.1% 90.4% 77.4% 59.0%
# 10 90.4% 81.7% 59.9% 34.9%
# 20 81.8% 66.8% 35.8% 12.2%
# 40 66.9% 44.6% 12.9% 1.5%Read along the 5% column. Three turns is fine, ten turns is a coin flip you win more often than not, forty turns almost never works. Nothing about the model changed between those rows. The number of turns is a reliability decision, and reading the table the other way says the only two levers are a lower per-turn error rate, which is slow to move, and fewer turns between checks, which is a Tuesday afternoon.
The second lever is the one worth quantifying, because it is the argument for checkpoints. Insert a verification step every few turns that catches most of the errors it looks for, and the run stops compounding: the length of the chain is no longer the whole run but the gap between checks.
import numpy as np
def clean_run_rate(n_turns, p_err, check_every=None, catch=0.9, trials=20000, seed=4):
"""A wrong fact at any turn poisons everything after it, unless a check
catches it. catch is the share of errors a verification step notices."""
rng = np.random.default_rng(seed)
clean = 0
for _ in range(trials):
poisoned = False
for turn in range(1, n_turns + 1):
if not poisoned and rng.random() < p_err:
poisoned = True
if poisoned and check_every and turn % check_every == 0:
if rng.random() < catch:
poisoned = False # caught and corrected here
clean += not poisoned
return clean / trials
for n in (10, 20, 40):
bare = clean_run_rate(n, 0.05)
every5 = clean_run_rate(n, 0.05, check_every=5)
every2 = clean_run_rate(n, 0.05, check_every=2)
print(f"{n:>3} turns at 5% per turn: unattended {bare:>6.1%} "
f"checked every 5 {every5:>6.1%} checked every 2 {every2:>6.1%}")
# 10 turns at 5% per turn: unattended 60.3% checked every 5 97.7% checked every 2 98.9%
# 20 turns at 5% per turn: unattended 35.9% checked every 5 97.5% checked every 2 98.9%
# 40 turns at 5% per turn: unattended 12.9% checked every 5 97.6% checked every 2 99.0%That is the result to carry around. Unattended, reliability falls off a cliff with length. Checked every five turns, a forty-turn run is as reliable as a ten-turn one, and going from every five to every two buys almost nothing for two and a half times the checking. So the useful question is not how long the run is but how many turns pass between one verification and the next, and the next section is about building those checks.
Why long agent runs are riskier than long conversations
In a conversation a person notices drift and corrects it. In an unattended agent run nobody is watching, so the error compounds unchecked. This is the strongest argument for short runs with checkpoints rather than one long autonomous one.
Where people go wrong
Believing the agent's own account of what it did. It is generated text, produced by the same process that produced the mistake, and it is the least reliable part of the output.
Build this: Give an agent a four-part task and a tool that silently fails on the fourth. Check whether the final summary claims success, then add a verification step that catches it.
Open this section on its own pageDesigning so failure is survivable
By the end: Your agent degrades rather than collapses, and mistakes are recoverable.
- Idempotent tools
- Dry run and approval gates
- Checkpointing and resuming
- Verification as a step
You can name the failures; this is how you make them survivable. The aim is not an agent that never goes wrong, because that is not on offer. It is an agent whose mistakes can be retried without doing the thing twice, stopped before anything irreversible, resumed from where a long run died, and checked before it claims to be finished. Each is a small piece of engineering, and their absence is what turns a bad run into an incident with a manual cleanup at the end of it.
Idempotency#
A retried tool call must not do the thing twice. This matters more with agents than with ordinary code, because retries happen for reasons you did not plan: a timeout where the action actually succeeded, a resumed run, a model repeating itself.
def issue_refund(order_id: str, idempotency_key: str):
if already_processed(idempotency_key):
return previous_result(idempotency_key) # safe to call again
result = payments.refund(order_id)
record(idempotency_key, result)
return resultThe cost of getting this wrong is not abstract, so here it is with the money counted. The payment provider below always applies the charge and sometimes fails to return a response, which is the ordinary behaviour of a network. The naive version treats the timeout as a failure and tries again.
LEDGER = [] # every charge the payment system actually made
PROCESSED = {} # idempotency key -> the result we returned
def payments_refund(order_id, cents, flaky):
"""The charge lands. Whether the response comes back is another matter."""
LEDGER.append((order_id, cents))
if flaky:
raise TimeoutError("no response from the payment provider")
return {"refund_id": "R-" + str(len(LEDGER)), "cents": cents}
def refund_naive(order_id, cents, flaky):
try:
return payments_refund(order_id, cents, flaky)
except TimeoutError:
return payments_refund(order_id, cents, False) # "it failed, try again"
def refund_idempotent(order_id, cents, key, flaky):
if key in PROCESSED:
return PROCESSED[key]
try:
result = payments_refund(order_id, cents, flaky)
except TimeoutError:
# Unknown outcome. Ask the payment system what happened, do not retry.
prior = [c for o, c in LEDGER if o == order_id]
result = {"refund_id": "recovered", "cents": prior[-1]} if prior else None
if result is None:
raise
PROCESSED[key] = result
return result
for label, fn in (("naive retry", lambda: refund_naive("A-1", 2900, flaky=True)),
("idempotent", lambda: refund_idempotent("A-2", 2900,
key="run-77:refund:A-2",
flaky=True))):
LEDGER.clear(); PROCESSED.clear()
fn()
charged = sum(c for _, c in LEDGER)
print(f"{label:<14}{len(LEDGER)} charges {charged} cents {LEDGER}")
# the same call arriving twice, which is what a resumed run looks like
LEDGER.clear(); PROCESSED.clear()
for _ in range(3):
refund_idempotent("A-3", 2900, key="run-77:refund:A-3", flaky=False)
print(f"{'replayed x3':<14}{len(LEDGER)} charges {sum(c for _, c in LEDGER)} cents")
# naive retry 2 charges 5800 cents [('A-1', 2900), ('A-1', 2900)]
# idempotent 1 charges 2900 cents [('A-2', 2900)]
# replayed x3 1 charges 2900 centsTwo details make this work in practice. The key has to be derived, not generated: something like the run id, the tool name and a hash of the arguments, so that the same intent in the same run produces the same key while a second genuine refund on the same order produces a different one. Generating a fresh uuid per call is the common mistake and defeats the whole mechanism, because the retry then carries a key nobody has seen.
And the recovery path matters more than the cache. A timeout is an unknown outcome, so the right move is to ask the downstream system what it has, by idempotency key if it supports them and by a search for a matching recent record if it does not. Retrying blind is only safe when the operation is naturally idempotent, which is to say when it sets a value rather than changing one: setting a status to refunded is safe to repeat, adding 29.00 to a balance is not.
Dry run#
Let the agent produce a plan of actions without executing them. Useful in development, useful in testing, and useful in production when a person should see what is about to happen.
DRY_RUN = True
def refund_tool(order_id, amount, idempotency_key):
if DRY_RUN:
return {"ok": True, "would_have": f"refunded {order_id} for {amount}"}
return refund_idempotent(order_id, amount, idempotency_key, flaky=False)
print(refund_tool("A-4471", 2900, "run-77:refund:A-4471"))
# {'ok': True, 'would_have': 'refunded A-4471 for 2900'}Approval gates#
Anything irreversible or externally visible should pause for a person. The agent proposes, a human confirms, then it executes. Choose the threshold deliberately: everything needing approval defeats the purpose, and nothing needing it is how money moves without anyone deciding.
- Always gate: sending money, deleting data, anything that reaches a customer.
- Gate above a threshold: refunds over an amount, bulk operations over a count.
- Do not gate: reads, searches, drafts, anything reversible in one click.
That threshold is usually argued about on instinct, and it does not have to be. Write down the traffic, write down what a wrong action costs to clean up, and the three policies separate immediately. The harm figures below are the part to argue about; the arithmetic on top of them is not.
# One thousand actions from a week of traffic, with what each one costs to
# clean up if the agent gets it wrong. The harm figures are yours to argue
# about; the arithmetic is not.
ACTIONS = [
# kind, count, external, cents, bulk, harm_if_wrong
("search_orders", 620, False, 0, 0, 0),
("get_customer", 180, False, 0, 0, 0),
("draft_reply", 60, False, 0, 0, 0),
("send_email", 70, True, 0, 0, 40),
("refund_small", 45, False, 900, 0, 9),
("refund_large", 8, False, 24000, 0, 240),
("bulk_close", 14, False, 0, 120, 600),
("delete_account", 3, False, 0, 0, 2000),
]
P_WRONG = 0.03 # how often the agent picks the wrong target
def gate_none(a): return False
def gate_all(a): return a[5] >= 0
def gate_rule(a):
kind, _, external, cents, bulk, _ = a
if kind == "delete_account": return True # always: irreversible
if external: return True # always: reaches a customer
if cents > 5000: return True # threshold: large amounts
if bulk > 50: return True # threshold: bulk operations
return False
print(f"{'policy':<14}{'approvals':>11}{'harm left':>12}{'harm stopped':>14}"
f"{'per approval':>14}")
baseline = sum(n * P_WRONG * harm for _, n, _, _, _, harm in ACTIONS)
for name, policy in (("gate nothing", gate_none), ("gate everything", gate_all),
("by rule", gate_rule)):
approvals = sum(n for a in ACTIONS for n in [a[1]] if policy(a))
left = sum(n * P_WRONG * harm for a in ACTIONS
for _, n, _, _, _, harm in [a] if not policy(a))
stopped = baseline - left
per = stopped / approvals if approvals else float("nan")
print(f"{name:<14}{approvals:>11}{left:>11.0f} {stopped:>13.0f}{per:>14.2f}")
print()
print("gated by rule:", [a[0] for a in ACTIONS if gate_rule(a)])
# policy approvals harm left harm stopped per approval
# gate nothing 0 586 0 nan
# gate everything 1000 0 586 0.59
# by rule 95 12 574 6.04
#
# gated by rule: ['send_email', 'refund_large', 'bulk_close', 'delete_account']The last column is the one to take to the argument, because gating everything is not the safe option, it is the option where approvals stop being read. A reviewer asked to approve a thousand searches approves the thousand-and-first thing without looking, and the delete arrives in that queue. Reserving the interruption for the ninety-five actions that carry the harm is what keeps the review real.
In code, the gate belongs in the dispatcher rather than inside each tool, so that a new tool is gated by default rather than by remembering. The shape that works is a policy function returning allow, deny or needs-approval; the loop pauses on needs-approval and persists the pending action, because a human is not going to answer within the request. That means an approval gate implies durable state, which is the same state a resumable run needs, and the two are usually built together.
Checkpoint and resume#
Save state after each turn so a crash loses one turn rather than the whole run. The subtlety is that the message list is not the whole state: if a tool changed something in the world, replaying that turn repeats the change, which is where idempotency earns its place.
CHECKPOINTS = {}
def step(messages):
return {"turn": len(messages)} # stands in for the model call
def apply(response, messages):
return messages + [response]
def save_checkpoint(run_id, turn, messages):
CHECKPOINTS[run_id] = (turn, list(messages))
run_id, messages, max_turns = "run-77", [], 4
for turn in range(max_turns):
response = step(messages)
messages = apply(response, messages)
save_checkpoint(run_id, turn, messages) # after, not before
print(CHECKPOINTS["run-77"][0], "turns saved,", len(CHECKPOINTS["run-77"][1]), "messages")
# 3 turns saved, 4 messagesSaving after the turn rather than before is the right choice and it is not sufficient, which is worth seeing rather than being told. There is a window between the side effect landing and the checkpoint being written, and no ordering closes it: whatever you write first, the process can die in between. The run below is a five-step plan killed at step three, resumed from whatever the checkpoint said.
PLAN = ["refund A-9", "email ada", "close T-9", "note in CRM", "update forecast"]
class Crash(Exception):
pass
def attempt(start, effects, seen, idempotent, save, crash_at=None, in_window=False):
"""Runs the plan from 'start'. A crash in the window happens after the side
effect and before the checkpoint is written, which is the gap no ordering
can close."""
saved = start
for i in range(start, len(PLAN)):
if save == "before":
saved = i
key = "run-77:" + PLAN[i]
if not (idempotent and key in seen):
effects.append(PLAN[i])
seen.add(key)
if crash_at == i and in_window:
raise Crash(saved)
if save == "after":
saved = i + 1
if crash_at == i:
raise Crash(saved)
return saved
def run(save, idempotent, in_window):
effects, seen = [], set()
try:
attempt(0, effects, seen, idempotent, save, crash_at=2, in_window=in_window)
except Crash as c:
attempt(c.args[0], effects, seen, idempotent, save)
dupes = sorted({s for s in effects if effects.count(s) > 1})
return f"{len(effects)} effects {dupes}"
print(f"{'policy':<30}{'crash between steps':>24}{'crash in the window':>24}")
for label, save, idem in (("checkpoint before the step", "before", False),
("checkpoint after the step", "after", False),
("after, and idempotent", "after", True)):
print(f"{label:<30}{run(save, idem, False):>24}{run(save, idem, True):>24}")
print("five steps, so five effects is correct")
# policy crash between steps crash in the window
# checkpoint before the step 6 effects ['close T-9'] 6 effects ['close T-9']
# checkpoint after the step 5 effects [] 6 effects ['close T-9']
# after, and idempotent 5 effects [] 5 effects []
# five steps, so five effects is correctOnly the last row is correct in both columns, and it is correct because of the previous section rather than this one. Checkpointing decides how much work a crash costs you; idempotency decides whether the recovery is safe. Neither substitutes for the other, and the pair is what makes a long run resumable at all.
What to put in the checkpoint is the other half. The message list is the obvious part and the least interesting: also save the run id, the budget consumed so far, the idempotency keys already used with their results, any pending approval, and a version string for the prompt and tool definitions. That last one stops a run resuming into a different agent after a deploy, which produces a transcript that makes no sense to anybody reading it afterwards.
Detecting a stuck loop, and what each detector misses#
The agent-loop topic puts a repeated-call detector in the loop and shows what it saves. The question left open there is how much of the problem it actually catches, because a stuck agent does not always repeat itself exactly. Four kinds of stuck run and four detectors, over four hundred runs of each kind, answer it. The healthy runs include a legitimate retry about a quarter of the time, which is the realistic thing that makes this hard.
import numpy as np
N_TURNS = 16
def make_run(kind, rng):
"""Returns a list of (tool, args, gained_a_new_fact) for one run."""
start = int(rng.integers(2, 6))
turns = [("get_order", {"id": "A-" + str(t)}, True) for t in range(start)]
for t in range(start, N_TURNS):
i = t - start
if kind == "exact":
turns.append(("search", {"q": "refund policy"}, False))
elif kind == "reworded":
q = ["refund policy", "Refund Policy", "policy refund",
"REFUND POLICY"][i % 4]
turns.append(("search", {"q": q}, False))
elif kind == "cycle":
turns.append(("search", {"q": "refund"}, False) if i % 2 == 0
else ("read_doc", {"id": "POL-14"}, False))
elif kind == "drift":
turns.append(("search", {"q": "refund question " + str(i)}, False))
else: # healthy, with a legitimate retry
retry = rng.random() < 0.25
turns.append(("get_order", {"id": "A-" + str(t - retry)}, not retry))
return turns
def canon(args):
return tuple(sorted(" ".join(sorted(str(v).lower().split())) for v in args.values()))
def detect(turns, mode, window=6, patience=4):
"""First turn at which this detector fires, or None."""
seen, last, repeats, since_progress = [], None, 0, 0
for t, (name, args, gained) in enumerate(turns):
sig = (name, tuple(sorted(args.items()))) if mode in ("exact", "window") \
else (name, canon(args))
if mode in ("exact", "canonical"):
repeats = repeats + 1 if sig == last else 0
last = sig
if repeats >= 2:
return t
elif mode == "window":
if sig in seen[-window:]:
return t
seen.append(sig)
else: # no progress
since_progress = 0 if gained else since_progress + 1
if since_progress >= patience:
return t
return None
rng = np.random.default_rng(0)
KINDS = ("exact", "reworded", "cycle", "drift", "healthy")
MODES = ("exact", "canonical", "window", "progress")
RUNS = {k: [make_run(k, rng) for _ in range(400)] for k in KINDS}
print(f"{'detector':<12}" + "".join(f"{k:>11}" for k in KINDS))
for mode in MODES:
cells = []
for kind in KINDS:
hits = [detect(r, mode) for r in RUNS[kind]]
rate = sum(h is not None for h in hits) / len(hits)
cells.append(f"{rate:>10.0%} ")
print(f"{mode:<12}" + "".join(cells))
# detector exact reworded cycle drift healthy
# exact 100% 0% 0% 0% 0%
# canonical 100% 100% 0% 0% 0%
# window 100% 100% 100% 0% 98%
# progress 100% 100% 100% 100% 2%Read the table as a progression and it stops being obvious. Exact argument equality catches one stuck pattern in four. Canonicalising the arguments, twenty minutes of work, doubles that. A window catches cycles as well, and is unusable: an agent that reads the same order twice in six turns is not stuck, it is working, and a detector that stops those runs is worse than no detector. Only the progress check catches the drifting agent, the one issuing a genuinely different search every turn and learning nothing, which is the hardest kind to see in a transcript and the most common in practice.
So define progress explicitly instead of inferring it from the calls. Something is progress if it puts a new fact in the run's ledger: a record retrieved that was not there before, a side effect applied, an open question answered. Count the turns since the last one, and when the count passes a small number, intervene. The detector is a handful of lines, it does not care how the model phrases things, and it is the only one on that table you can leave switched on.
print("median turn the detector fires on a stuck run (16-turn runs):")
for mode in MODES:
fired = [detect(r, mode) for kind in ("exact", "reworded", "cycle", "drift")
for r in RUNS[kind]]
fired = [f for f in fired if f is not None]
print(f" {mode:<10}{np.median(fired):>5.1f} fired on {len(fired)} of 1600 stuck runs")
# exact 6.0 fired on 400 of 1600 stuck runs
# canonical 6.0 fired on 800 of 1600 stuck runs
# window 6.0 fired on 1200 of 1600 stuck runs
# progress 7.0 fired on 1600 of 1600 stuck runsVerify before reporting success#
Add an explicit verification step that checks the world rather than asking the model whether it finished. It is cheap, it catches partial completion, and it converts a silent failure into a clear one.
from dataclasses import dataclass, field
@dataclass
class Requirement:
name: str
check: callable
@dataclass
class Task:
requirements: list = field(default_factory=list)
@dataclass
class AgentResult:
complete: bool
missing: list
STATE = {"refunded": True, "emailed": True, "crm_note": False}
task = Task([Requirement("order refunded", lambda: STATE["refunded"]),
Requirement("customer emailed", lambda: STATE["emailed"]),
Requirement("reason in CRM", lambda: STATE["crm_note"])])
def verify(task) -> tuple[bool, list[str]]:
missing = [r.name for r in task.requirements if not r.check()]
return (not missing), missing
ok, missing = verify(task)
print(AgentResult(complete=ok, missing=missing) if not ok else "verified")
# AgentResult(complete=False, missing=['reason in CRM'])Never let the model grade its own completion
Asking the agent whether it finished uses the same process that produced the mistake, and it will say yes. Verification has to be code checking real state.
Where people go wrong
Checkpointing before executing the turn rather than after. On resume the turn runs a second time, and without idempotency the side effect happens twice.
Build this: Add idempotency keys, checkpointing and a verification step to an agent. Then kill the process mid-run, resume it, and confirm nothing happened twice and the verification still passes.
Open this section on its own pageWhere you use this
A topic is easier to care about when you can see what it is for. These projects lean on it at a named stage.
- Stage 4, Make failure survivable: The specific ways this loop falls over once a tool is unreliable, so you can reproduce them on purpose.
Now actually do it
Reading a topic is not learning it. These are the hours that count, and each one tells you when you are finished.
1Catch a partial completion
4hGive an agent a four-part task with a tool that silently fails on the fourth. Check whether the summary claims success. Then add a verification step that checks the world.
Done when: The agent now reports incomplete when it is incomplete, verified in code rather than by asking the model.
2Survive a crash mid-run
4hAdd idempotency keys and checkpointing after each turn. Kill the process mid-run, resume, and confirm nothing happened twice.
Done when: A resumed run completes correctly and the side effects occurred exactly once.
8 hours of practice
Words used here
1 term this topic teaches, each with a one-line definition if one of them stopped you.
- Hallucination
- A fluent, confident statement that is not true — an invented citation, a function that does not exist, a plausible number.
Interview questions
6 questions, grouped by the section that teaches the answer. An answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
How agents fail
What are the failure modes of an agent that ordinary software does not have?
Partial completion reported as success, confidently taking the wrong action, and compounding error where a small mistake early becomes an assumption later. None of these raise, so logs alone will not tell you.
How do you detect partial completion?
Do not trust the agent's summary. Check the world: whether each intended side effect actually happened. This is why agent evals assert on end state rather than on the text of the final answer.
Why does error compound in an agent?
Every turn is conditioned on all previous turns, so a wrong intermediate result becomes an assumption and then a conclusion. Without a human watching, nothing interrupts the chain, which is why shorter runs with checkpoints are safer than one long autonomous run.
Designing so failure is survivable
How do you make an agent safe to retry?
Idempotent tools keyed on run id and intent, so a repeated call returns the previous result instead of acting twice. Checkpoint after each turn so a resumed run continues rather than restarting, and remember that the message list is not the whole state when a tool changed the world.
Then they push further
The payment API does not support idempotency keys. What do you do?
Record intent before calling: write a pending record with a unique id and check for it before any retry. Resolve uncertain cases by asking the payment system what actually happened instead of calling again.
What gets marked up. Strong answers mention idempotency keys and checking state before retrying, and treat a timeout as an unknown outcome rather than a failure.
Which agent actions need human approval?
Anything irreversible or externally visible: money, deletion, anything reaching a customer. Threshold-gate the rest by amount or volume. Gating everything defeats the purpose and gating nothing is how unreviewed actions happen.
How do you confirm an agent actually completed a task?
Verify against the world in code, not by asking the model. Check that each required side effect happened. Asking the agent uses the same process that produced any mistake, so it will report success either way.
Resources
Free to watch and read, picked for this topic.
Videos
For how agents fail
For designing so failure is survivable
Full courses: the whole topic, start to finish
Stop 20 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.