How agents fail
By the end: You can name the ways an agent goes wrong and recognise each one in a trace.
- Failures that raise, and failures that do not
- Partial completion
- The confident wrong action
- Compounding error over turns
New to agent reliability? Agents fail in ways ordinary programs do not. A normal bug raises an error; an agent calls the same broken tool forty times, or acts confidently on something it got wrong, or stops halfway and reports success. Agent reliability is the set of defences against that: budgets on turns and spend, tools that are safe to run twice, dry runs, saved checkpoints so a failed run can resume, and a person asked to approve anything destructive. Why it matters
Ordinary software mostly fails loudly: something raises, something times out, and you get an alert. Agents do that too, and those are the easy cases. The failures that cost you are the ones where nothing errors at all — the run completes, the agent reports success, and the work is half done or confidently wrong. This section gives you a vocabulary for those failures and shows how each one looks in a trace, which is what you need before you can design against them.
Two categories#
Some agent failures look like ordinary software failures: a timeout, a rate limit, a tool raising. Those are the easy ones, because something tells you.
The dangerous category is the one where nothing errors. The agent finishes, reports success, and the work is wrong. Nothing in your logs says so.
There is a third category between them that is worth separating out, because people keep putting it in the first one. A tool call that times out has not failed. It has an unknown outcome: the request may have arrived, been processed and charged, and only the response gone missing. Treating a timeout as a failure and retrying is how a customer gets refunded twice, and it is the most common way money moves incorrectly in an agent system. The next section is largely about that.
Partial completion#
An agent asked to do four things does three, then summarises as though it did four. This is common and it is easy to miss, because the summary is confident and reads well.
Task: "Refund the order, email the customer, close the ticket,
and note the reason in the CRM."
Trace: refund_order ok
send_email ok
close_ticket ok
(no CRM call)
Answer: "Done. The customer has been refunded and notified,
the ticket is closed and the reason is recorded."The defence is not to trust the summary. Check the world: did the record actually change? That is also exactly what an agent eval should assert.
Worth running once, because the shape of it is more convincing than the description. Below, every tool returns ok and one of them writes nothing, which is what a stubbed integration, a silently-swallowed exception or a permissions problem all look like from inside the loop.
WORLD = {"orders": {"A-1": "shipped"}, "emails": [], "tickets": {"T-1": "open"},
"crm": {}}
def refund(order_id): WORLD["orders"][order_id] = "refunded"; return {"ok": True}
def send_email(to): WORLD["emails"].append(to); return {"ok": True}
def close_ticket(tid): WORLD["tickets"][tid] = "closed"; return {"ok": True}
def note_in_crm(cid, text):
return {"ok": True} # the bug: reports success, writes nothing
TOOLS = {"refund": refund, "send_email": send_email,
"close_ticket": close_ticket, "note_in_crm": note_in_crm}
CALLS = [("refund", ("A-1",)), ("send_email", ("ada@example.com",)),
("close_ticket", ("T-1",)), ("note_in_crm", ("C-118", "refunded, faulty"))]
results = [(name, TOOLS[name](*args)) for name, args in CALLS]
for name, result in results:
print(f"{name:<14}{result}")
SUMMARY = ("Done. The customer has been refunded and notified, the ticket is "
"closed and the reason is recorded in the CRM.")
REQUIREMENTS = {
"order refunded": lambda: WORLD["orders"]["A-1"] == "refunded",
"customer emailed": lambda: WORLD["emails"] == ["ada@example.com"],
"ticket closed": lambda: WORLD["tickets"]["T-1"] == "closed",
"reason in CRM": lambda: "C-118" in WORLD["crm"],
}
missing = [name for name, check in REQUIREMENTS.items() if not check()]
print()
print("the agent says:", SUMMARY)
print("every tool call returned ok:", all(r["ok"] for _, r in results))
print("requirements actually met :", len(REQUIREMENTS) - len(missing), "of",
len(REQUIREMENTS), " missing:", missing)
# refund {'ok': True}
# send_email {'ok': True}
# close_ticket {'ok': True}
# note_in_crm {'ok': True}
#
# the agent says: Done. The customer has been refunded and notified, the ticket is closed and the reason is recorded in the CRM.
# every tool call returned ok: True
# requirements actually met : 3 of 4 missing: ['reason in CRM']Write the requirements down as a dict of checks when you write the task, not afterwards. It costs a few minutes, it gives the agent eval its assertions for free, and it converts the most expensive failure mode in this section into a returned error.
The confident wrong action#
The agent refunds the wrong order, emails the wrong customer, or updates the wrong record. Everything succeeded technically. The damage is real and reversing it is a support problem rather than a code problem.
Compounding error#
Each turn builds on the previous ones. A small mistake at turn two becomes an assumption at turn three and a conclusion at turn six. By the end the agent is confidently working on a problem that does not exist.
This is arithmetic rather than folklore, and the arithmetic is worth knowing because it decides how long a run you can allow. If each turn has an independent chance of introducing something wrong, and a wrong fact poisons everything conditioned on it, the chance of a clean run is that per-turn success rate raised to the number of turns.
print("chance the whole run is clean, given a per-turn error rate")
print(f"{'turns':>6}" + "".join(f"{p:>10.0%}" for p in (0.01, 0.02, 0.05, 0.10)))
for n in (3, 5, 10, 20, 40):
print(f"{n:>6}" + "".join(f"{(1 - p) ** n:>10.1%}" for p in (0.01, 0.02, 0.05, 0.10)))
# chance the whole run is clean, given a per-turn error rate
# turns 1% 2% 5% 10%
# 3 97.0% 94.1% 85.7% 72.9%
# 5 95.1% 90.4% 77.4% 59.0%
# 10 90.4% 81.7% 59.9% 34.9%
# 20 81.8% 66.8% 35.8% 12.2%
# 40 66.9% 44.6% 12.9% 1.5%Read along the 5% column. Three turns is fine, ten turns is a coin flip you win more often than not, forty turns almost never works. Nothing about the model changed between those rows. The number of turns is a reliability decision, and reading the table the other way says the only two levers are a lower per-turn error rate, which is slow to move, and fewer turns between checks, which is a Tuesday afternoon.
The second lever is the one worth quantifying, because it is the argument for checkpoints. Insert a verification step every few turns that catches most of the errors it looks for, and the run stops compounding: the length of the chain is no longer the whole run but the gap between checks.
import numpy as np
def clean_run_rate(n_turns, p_err, check_every=None, catch=0.9, trials=20000, seed=4):
"""A wrong fact at any turn poisons everything after it, unless a check
catches it. catch is the share of errors a verification step notices."""
rng = np.random.default_rng(seed)
clean = 0
for _ in range(trials):
poisoned = False
for turn in range(1, n_turns + 1):
if not poisoned and rng.random() < p_err:
poisoned = True
if poisoned and check_every and turn % check_every == 0:
if rng.random() < catch:
poisoned = False # caught and corrected here
clean += not poisoned
return clean / trials
for n in (10, 20, 40):
bare = clean_run_rate(n, 0.05)
every5 = clean_run_rate(n, 0.05, check_every=5)
every2 = clean_run_rate(n, 0.05, check_every=2)
print(f"{n:>3} turns at 5% per turn: unattended {bare:>6.1%} "
f"checked every 5 {every5:>6.1%} checked every 2 {every2:>6.1%}")
# 10 turns at 5% per turn: unattended 60.3% checked every 5 97.7% checked every 2 98.9%
# 20 turns at 5% per turn: unattended 35.9% checked every 5 97.5% checked every 2 98.9%
# 40 turns at 5% per turn: unattended 12.9% checked every 5 97.6% checked every 2 99.0%That is the result to carry around. Unattended, reliability falls off a cliff with length. Checked every five turns, a forty-turn run is as reliable as a ten-turn one, and going from every five to every two buys almost nothing for two and a half times the checking. So the useful question is not how long the run is but how many turns pass between one verification and the next, and the next section is about building those checks.
Why long agent runs are riskier than long conversations
In a conversation a person notices drift and corrects it. In an unattended agent run nobody is watching, so the error compounds unchecked. This is the strongest argument for short runs with checkpoints rather than one long autonomous one.
Where people go wrong
Believing the agent's own account of what it did. It is generated text, produced by the same process that produced the mistake, and it is the least reliable part of the output.
If this comes up in an interview
What are the failure modes of an agent that ordinary software does not have?
Partial completion reported as success, confidently taking the wrong action, and compounding error where a small mistake early becomes an assumption later. None of these raise, so logs alone will not tell you.
How do you detect partial completion?
Do not trust the agent's summary. Check the world: whether each intended side effect actually happened. This is why agent evals assert on end state rather than on the text of the final answer.
Why does error compound in an agent?
Every turn is conditioned on all previous turns, so a wrong intermediate result becomes an assumption and then a conclusion. Without a human watching, nothing interrupts the chain, which is why shorter runs with checkpoints are safer than one long autonomous run.
Build this: Give an agent a four-part task and a tool that silently fails on the fourth. Check whether the final summary claims success, then add a verification step that catches it.
Watch it explained
This is one section of Agent reliability, 8 hours across 2 sections. Read it in full to get the whole topic in order.
Stop 20 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.