Evaluating agents and multi-step systems
By the end: You can evaluate a system that takes several steps, and locate which step a failure started in.
- Final outcome versus the path taken
- Checking the end state, not the transcript
- Finding the first failing step
- Tool call correctness
- Reliability across repeated runs
- Cost and step budgets
New to evaluation? An eval is a fixed set of test cases you run your system against, plus a rule for deciding whether each answer was good enough. It is a test suite for something that never gives the same answer twice: instead of checking the output matches a stored string, you check it satisfies what you asked for — the right fields present, the right document cited, the refund policy not invented. Why it matters
Evaluation gets harder as soon as a system takes more than one step. There is no single correct transcript to compare against, the same task can be done well by several different routes, and a mistake at step two may only become visible at step nine. This section is about what to check instead, which is mostly the state of the world at the end, and about tracing backwards to the step where a run first went wrong. Repeated runs and step budgets belong here too, because an agent that succeeds one time in three is not working.
Why agents need a different approach#
A single call has one input and one output. An agent makes a sequence of decisions, and two runs on the same task can take completely different routes. Judging the final message alone misses most of what matters, and judging every step against one expected path punishes an agent for finding a different valid route.
Check the outcome in the world#
Where the agent changes something, grade the change itself. If it was asked to cancel Jack's second order and issue a refund, look at the database afterwards. Is that order cancelled? Is the other order untouched? Is there exactly one refund of the right amount? This is more reliable than asking a judge whether the transcript sounds right.
def grade(env_before, env_after):
orders = env_after.orders_for("jack")
return all([
orders["A-102"].status == "cancelled",
orders["A-101"].status == env_before.orders_for("jack")["A-101"].status,
len(env_after.refunds_for("A-102")) == 1,
env_after.refunds_for("A-102")[0].amount == 49.00,
])Notice what the second line is doing. Checking that the target changed is the easy half; checking that nothing else did is the half that catches the expensive failures. An agent that cancels the right order and also cancels the other one passes any grader that only looks at what it was asked about.
This only works if each task starts from a known world and you can compare against it, which means a sandbox you can reset. Build it as a fixture: seed the database, run the task, diff the whole state, assert on the diff. Then a grader is the list of changes you expected, and everything else in the diff is a finding. Agents that touch the real world need the same thing with fakes at the boundary, because an eval suite that sends real emails will be run once.
Then look at the path#
Outcome checks tell you whether it worked. They do not tell you why it failed. For failing runs, walk the trace and mark the first step that went wrong. Over many runs, tally which step type fails first most often.
First failing step, 40 failed runs
understand request 6
choose tool 11
fill tool arguments 15
interpret tool result 5
final reply 3A table like this turns 'the agent is flaky' into 'most failures start when the agent fills in tool arguments'. That is fixable, often with a better tool description, a stricter argument schema, or a lookup tool so it stops guessing ids.
Most of that table can be produced by code rather than by reading. Walk the steps in order, apply the checks that do not need to know the task, and stop at the first one that trips. The checks below are generic enough to be worth writing once: a tool that does not exist, an argument of the wrong type, acting on an id the agent never looked up, and retrying an identical call that has already failed.
from collections import Counter
ALLOWED = {"lookup_order", "cancel_order", "issue_refund", "send_email"}
ARG_TYPES = {"order_id": str, "amount_cents": int, "reason": str}
def first_bad_step(trace):
"""Walk the steps and return (index, what went wrong), or None if the path
is clean. Only the first is reported: later steps inherit the mistake."""
seen_ids = set()
for i, step in enumerate(trace):
if step["tool"] not in ALLOWED:
return i, "tool does not exist"
for key, value in step["args"].items():
if key not in ARG_TYPES:
return i, "unknown argument " + key
if not isinstance(value, ARG_TYPES[key]):
return i, "argument " + key + " has the wrong type"
oid = step["args"].get("order_id")
if step["tool"] != "lookup_order" and oid is not None and oid not in seen_ids:
return i, "acted on an id it never looked up"
nxt = trace[i + 1] if i + 1 < len(trace) else None
if step["result"] == "not found" and nxt is not None:
if nxt["tool"] == step["tool"] and nxt["args"] == step["args"]:
return i + 1, "retried the identical failing call"
if step["tool"] == "lookup_order" and step["result"] != "not found":
seen_ids.add(oid)
return None
TRACES = [
[{"tool": "lookup_order", "args": {"order_id": "A-101"}, "result": "ok"},
{"tool": "cancel_order", "args": {"order_id": "A-101"}, "result": "ok"}],
[{"tool": "cancel_order", "args": {"order_id": "A-102"}, "result": "ok"}],
[{"tool": "lookup_order", "args": {"order_id": "A-103"}, "result": "not found"},
{"tool": "lookup_order", "args": {"order_id": "A-103"}, "result": "not found"}],
[{"tool": "lookup_order", "args": {"order_id": "A-104"}, "result": "ok"},
{"tool": "issue_refund", "args": {"order_id": "A-104", "amount_cents": "4900"},
"result": "ok"}],
[{"tool": "refund_order", "args": {"order_id": "A-105"}, "result": "error"}],
]
tally = Counter()
for t in TRACES:
bad = first_bad_step(t)
print(bad if bad else "clean")
if bad:
tally[bad[1]] += 1
print(tally.most_common())
# clean
# (0, 'acted on an id it never looked up')
# (1, 'retried the identical failing call')
# (1, 'argument amount_cents has the wrong type')
# (0, 'tool does not exist')
# [('acted on an id it never looked up', 1), ('retried the identical failing call', 1),
# ('argument amount_cents has the wrong type', 1), ('tool does not exist', 1)]Run that over every failing trace and the tally is your first-failing-step table, built automatically and updated on every eval run. What it cannot judge is whether the agent chose the right tool for this particular request, which still needs either an expected-tool annotation on the task or a person reading. Automate the checks that do not need the task, and spend the reading time on the ones that do.
Tool calls are checkable#
- Did it call a tool at all when one was needed, and not call one when none was needed?
- Was it the right tool?
- Do the arguments pass the schema, and are they the right values for this request?
- Did it retry sensibly after an error, or loop on the same failing call?
Most of these are code checks against the trace. Save judges for questions like whether the reply correctly explained what happened.
Run it more than once#
An agent that succeeds on a task half the time is not half useful. For a customer who hits it three times, it fails at least once almost nine times in ten. Run each task several times and report two numbers: the share of runs that succeed, and the share of tasks where every run succeeds. The second is what users experience as reliability.
# Ten tasks, four runs each. 1 means the end state was correct.
results = {
"cancel one of two orders": [1, 1, 1, 1],
"refund a partial amount": [1, 0, 1, 1],
"change the delivery address": [1, 1, 1, 1],
"apply a discount code": [1, 1, 0, 1],
"split an order in two": [0, 1, 0, 1],
"close the account": [1, 1, 1, 1],
"reissue an invoice": [1, 1, 1, 0],
"move a subscription to annual": [1, 1, 1, 1],
"merge two accounts": [0, 0, 1, 0],
"extend a trial": [1, 1, 1, 1],
}
k = 4
avg_success = sum(sum(r) for r in results.values()) / (k * len(results))
all_k_pass = sum(all(r) for r in results.values()) / len(results)
print(f"per run {avg_success:.0%} all {k} runs {all_k_pass:.0%}")
for task, runs in results.items():
if not all(runs) and any(runs):
print(f" intermittent: {task} {runs}")
# per run 80% all 4 runs 50%
# intermittent: refund a partial amount [1, 0, 1, 1]
# intermittent: apply a discount code [1, 1, 0, 1]
# intermittent: split an order in two [0, 1, 0, 1]
# intermittent: reissue an invoice [1, 1, 1, 0]
# intermittent: merge two accounts [0, 0, 1, 0]Where this measure comes from
Reporting the chance that all k attempts succeed, often written pass^k, was popularised by the tau-bench benchmark for tool-using agents. It is a useful habit on your own tasks even if you never touch the benchmark.
Four runs is few enough that a task passing all four is weaker evidence than it feels. The left table below is what to expect from a task of a given real reliability; the right one is why agents with many steps are hard, since a run only succeeds if every step does.
print("chance a task passes every one of k runs, by its real per-run rate")
print(f"{'real rate':>10}" + "".join(f"{'k=' + str(k):>8}" for k in (1, 4, 10)))
for p in (0.99, 0.95, 0.90, 0.80, 0.70):
print(f"{p:>10.0%}" + "".join(f"{p ** k:>8.0%}" for k in (1, 4, 10)))
print()
print("a run of s steps, each independently right with probability q")
print(f"{'q':>6}" + "".join(f"{'s=' + str(s):>8}" for s in (5, 10, 20, 40)))
for q in (0.99, 0.98, 0.95, 0.90):
print(f"{q:>6}" + "".join(f"{q ** s:>8.0%}" for s in (5, 10, 20, 40)))
# chance a task passes every one of k runs, by its real per-run rate
# real rate k=1 k=4 k=10
# 99% 99% 96% 90%
# 95% 95% 81% 60%
# 90% 90% 66% 35%
# 80% 80% 41% 11%
# 70% 70% 24% 3%
#
# a run of s steps, each independently right with probability q
# q s=5 s=10 s=20 s=40
# 0.99 95% 90% 82% 67%
# 0.98 90% 82% 67% 45%
# 0.95 77% 60% 36% 13%
# 0.9 59% 35% 12% 1%The right-hand table also says where to spend effort. Going from 95% to 98% per step more than doubles the success rate of a twenty-step task. Adding a recovery path, so a failed step is retried with the error fed back rather than ending the run, changes the exponent rather than the base and is usually the larger win. Both are measurable with the same harness.
Budgets are part of correctness#
An agent that solves the task in forty steps and two dollars may be a failure in production. Record steps, tokens, cost and wall time per run, and set limits. A run that exceeds the budget counts as a failure, whatever the final answer was.
import numpy as np
rng = np.random.default_rng(21)
RUNS = 200
steps = rng.integers(4, 9, RUNS)
stuck = rng.random(RUNS) < 0.12 # runs that thrash before finishing
steps = np.where(stuck, steps + rng.integers(10, 30, RUNS), steps)
cost = steps * 0.004 + rng.normal(0, 0.0015, RUNS).clip(0)
correct = rng.random(RUNS) < 0.88
BUDGET_STEPS, BUDGET_COST = 15, 0.08
over = (steps > BUDGET_STEPS) | (cost > BUDGET_COST)
print(f"outcome correct {correct.mean():.0%}")
print(f"within step and cost budget {(~over).mean():.0%}")
print(f"both {(correct & ~over).mean():.0%}")
print(f"median steps {int(np.median(steps))}, p95 steps {int(np.percentile(steps, 95))}")
print(f"median cost {np.median(cost):.3f}, p95 cost {np.percentile(cost, 95):.3f}, "
f"mean cost {cost.mean():.3f}")
print(f"the {over.mean():.0%} of runs over budget spend "
f"{100 * cost[over].sum() / cost.sum():.0f}% of the money")
# outcome correct 90%
# within step and cost budget 88%
# both 78%
# median steps 7, p95 steps 24
# median cost 0.028, p95 cost 0.098, mean cost 0.035
# the 12% of runs over budget spend 36% of the moneyQuote the p95, never the mean, for steps and latency. The mean hides exactly the runs that hurt: here half the money goes on an eighth of the runs, and those are also the ones a user waits through. Setting the step limit at the p95 of successful runs is a reasonable starting point, and then every run that hits the limit is either a loop to fix or a task that genuinely needs a bigger budget.
Where people go wrong
Scoring an agent on one run per task. Variance between runs is large, and a single pass hides a task that fails half the time.
If this comes up in an interview
How would you evaluate a customer support agent that can issue refunds?
Build tasks in a sandboxed copy of the systems it touches, and grade the end state: the right order changed, nothing else changed, the refund amount is correct. Separately check tool calls in the trace and have a validated judge assess the reply. Run each task several times and report both per run success and the share of tasks that pass every time.
Then they push further
It passes 85% of runs. Is it ready to ship?
Not on that number alone. I would want the all-runs-pass rate, which could be far lower, the failure modes of the 15%, and whether any failure is costly, such as refunding the wrong order. A system that fails harmlessly at 15% may be fine; one that occasionally issues wrong refunds is not.
How do you stop the eval from teaching the agent to game the checks?
Keep a held-out task set nobody tunes against, check side effects as well as the target change so shortcuts are caught, and read a sample of passing traces, not just failing ones.
What gets marked up. Strong answers grade the resulting state rather than the transcript, mention repeated runs, and bring up the cost of each failure type. Weak answers describe only an LLM judge reading the conversation.
Why is final-answer accuracy not enough for an agent?
It does not tell you where failures start, it ignores side effects on other data, and it ignores cost and step count. Two runs can reach the same answer, one cleanly and one after ten wasted tool calls.
What does it mean to find the first failing step, and why bother?
Walk each failed trace and mark the earliest step that went wrong. Tallied across runs, it shows which kind of step fails most, such as filling tool arguments. That points directly at the fix, where 'the agent is unreliable' does not.
Build this: Take an agent you built. Write ten tasks with state-based graders, run each four times, and report per-run success, all-four success, and a first-failing-step table for the failures.
Watch it explained
This is one section of Evaluation, 35 hours across 7 sections. Read it in full to get the whole topic in order.
Stop 30 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.