Knowing whether it works
By the end: You have a number that tells you whether a change to the agent helped.
- Defining success when the path matters
- A fixed task set run on every change
- Tracking cost and turns alongside success
- The regression that hides behind a passing score
New to the agent loop? Agent is a loaded word for something small. Call the model; if the reply asks to use a tool, run the tool, add the result to the conversation, and call the model again. Keep going until it stops asking or until you decide it has had enough turns. That loop, a few dozen lines of ordinary code, is the agent — there is no other machinery hiding behind the word. Why it matters
You can build an agent and stop it running away. What you cannot yet do is say whether last week's prompt edit made it better. This section is about getting a number: what success means when the job is to change something in the world rather than produce text, and why turns and cost belong next to the pass rate. Without it, every later change to the agent is a guess nobody can check.
Watching is not measuring#
Agents are compelling to watch. You see it reason, call a tool, recover, and conclude, and it feels like evidence. It is not. You are seeing one run of a non-deterministic system, chosen by you, at a moment when you were hoping it would work.
This is where most agent projects stall. The demo is impressive, nobody can say whether last week's change helped, and eventually nobody wants to touch the prompt.
Defining success#
For an agent, the final answer is not the only thing that matters. A run that reaches the right answer after fourteen turns and three wrong tool calls is not a success in any useful sense.
- Outcome: is the end state correct? Was the record actually updated, the file actually written?
- Efficiency: how many turns and how much money did it take?
- Path: did it use sensible tools, or stumble into the answer?
Outcome and efficiency are easy to write down. Path is the one people either skip or overdo. The overdone version pins an exact sequence of tool calls and breaks the moment the agent finds a better route, which teaches you nothing except that it changed. The useful version is weaker and survives: assert that the required tool was called at all, that no forbidden tool was called, and that nothing was called more than once that should have been called once. Those three catch the cases you care about, and a run that reaches the right answer by a route you did not anticipate still passes.
from __future__ import annotations
from dataclasses import dataclass
from typing import Callable
@dataclass
class Task:
prompt: str
check: Callable[[AgentResult], bool]
max_turns: int = 8
max_usd: float = 0.20
TASKS = [
Task(
prompt="Refund order 4471 and email the customer.",
check=lambda r: order(4471).status == "refunded" and emails_sent() == 1,
),
...
]Check side effects, not wording
Asserting on the agent's closing sentence is fragile and misses the point. The question is whether the refund happened and whether exactly one email went out. Check the world.
Run it on every change#
import asyncio
from statistics import mean
async def evaluate(tasks):
results = await asyncio.gather(*(run_one(t) for t in tasks))
print(f"passed: {sum(r.passed for r in results)}/{len(results)}")
print(f"avg turns: {mean(r.turns for r in results):.1f}")
print(f"avg cost: ${mean(r.cost for r in results):.3f}")
print(f"total: ${sum(r.cost for r in results):.2f}")Twenty tasks is enough to be useful. Include the failures you have already seen in production, because those are the cases most likely to come back.
A harness, small enough to read#
The pieces are a world you can reset, tools that change it, tasks that assert on it afterwards, and three versions of the same agent to compare. Each version is written out as the calls it makes, so the comparison runs offline and reports the same numbers every time.
import copy
WORLD0 = {"orders": {"A-1": "shipped", "A-2": "shipped", "A-3": "shipped"},
"emails": [], "tickets": {"T-1": "open", "T-2": "open", "T-3": "open"}}
def refund(w, order_id): w["orders"][order_id] = "refunded"; return {"ok": True}
def send_email(w, to): w["emails"].append(to); return {"ok": True}
def close_ticket(w, tid): w["tickets"][tid] = "closed"; return {"ok": True}
def get_order(w, order_id): return {"status": w["orders"][order_id]}
TOOLS = {"refund": refund, "send_email": send_email,
"close_ticket": close_ticket, "get_order": get_order}
TASKS = [
{"id": "refund-and-notify",
"check": lambda w: w["orders"]["A-1"] == "refunded"
and w["emails"] == ["ada@example.com"]},
{"id": "refund-and-close",
"check": lambda w: w["orders"]["A-2"] == "refunded"
and w["tickets"]["T-2"] == "closed"},
]
# Three versions of the same agent, written out as the calls each one makes.
VERSIONS = {
"v1 baseline": {
"refund-and-notify": [("refund", {"order_id": "A-1"}),
("send_email", {"to": "ada@example.com"})],
"refund-and-close": [("refund", {"order_id": "A-2"}),
("close_ticket", {"tid": "T-2"})],
},
"v2 after prompt edit": {
"refund-and-notify": [("get_order", {"order_id": "A-1"}),
("refund", {"order_id": "A-1"}),
("get_order", {"order_id": "A-1"}),
("send_email", {"to": "ada@example.com"}),
("get_order", {"order_id": "A-1"})],
"refund-and-close": [("get_order", {"order_id": "A-2"}),
("refund", {"order_id": "A-2"}),
("get_order", {"order_id": "A-2"}),
("close_ticket", {"tid": "T-2"}),
("get_order", {"order_id": "A-2"})],
},
"v3 chattier": {
"refund-and-notify": [("refund", {"order_id": "A-1"}),
("send_email", {"to": "ada@example.com"}),
("send_email", {"to": "ada@example.com"})],
"refund-and-close": [("refund", {"order_id": "A-2"}),
("close_ticket", {"tid": "T-2"})],
},
}
IN_PER_M, OUT_PER_M = 3.00, 15.00
def cost_of(turns):
"""Same growing-context model as the budgets section."""
return sum((1200 + 700 * t) / 1e6 * IN_PER_M + 120 / 1e6 * OUT_PER_M
for t in range(turns))
def evaluate(version):
passed, turns, cost = 0, 0, 0.0
for task in TASKS:
world = copy.deepcopy(WORLD0) # every task starts from the same world
calls = VERSIONS[version][task["id"]]
for name, args in calls:
TOOLS[name](world, **args)
passed += bool(task["check"](world))
turns += len(calls) + 1 # one more model call to finish
cost += cost_of(len(calls) + 1)
return passed, turns, cost
print(f"{'version':<22}{'passed':>8}{'turns':>7}{'cost':>9}")
for version in VERSIONS:
passed, turns, cost = evaluate(version)
print(f"{version:<22}{passed:>5}/{len(TASKS)}{turns:>7} ${cost:>6.3f}")
# version passed turns cost
# v1 baseline 2/2 6 $ 0.045
# v2 after prompt edit 2/2 12 $ 0.128
# v3 chattier 1/2 7 $ 0.057Three things in that table are worth taking separately. Version two is the hidden regression: it passes exactly what the baseline passed, so a pass-rate-only eval reports no change, while it takes twice the turns and costs 0.128 against 0.045. Version three fails a task the baseline passed, and the reason it fails is the thing to notice.
The hidden regression#
Success rate flat, cost doubled
A prompt change can keep every task passing while doubling the turns taken. Nothing fails, nobody notices, and the bill quietly grows. This is exactly why cost and turn count belong in the eval output next to the pass rate.
Version three sent the customer two identical emails. It fails only because the check compares the list of emails to exactly one address. Write the same check the way most people write it and the regression disappears.
LOOSE = {"refund-and-notify": lambda w: w["orders"]["A-1"] == "refunded"
and len(w["emails"]) >= 1,
"refund-and-close": lambda w: w["orders"]["A-2"] == "refunded"
and w["tickets"]["T-2"] == "closed"}
def evaluate_with(version, checks):
passed = 0
for task in TASKS:
world = copy.deepcopy(WORLD0)
for name, args in VERSIONS[version][task["id"]]:
TOOLS[name](world, **args)
passed += bool(checks[task["id"]](world))
return passed
exact = {t["id"]: t["check"] for t in TASKS}
for version in VERSIONS:
print(f"{version:<22} exact check {evaluate_with(version, exact)}/2"
f" 'at least one email' check {evaluate_with(version, LOOSE)}/2")
# v1 baseline exact check 2/2 'at least one email' check 2/2
# v2 after prompt edit exact check 2/2 'at least one email' check 2/2
# v3 chattier exact check 1/2 'at least one email' check 2/2Agents make that mistake expensive in a way single completions do not, because doing something twice is one of their characteristic failures and a check written as at-least-once cannot see it. Assert on equality wherever you can: the list of emails, the number of rows written, the set of records touched. Where equality is genuinely too strict, assert on the count and on what was not touched, which catches an agent that refunded the right order and also the one next to it.
Two practical notes on building the harness. Run the tasks against a fresh copy of the world each time, or a task that mutates something makes every later task depend on the order they ran in, and a flaky eval is worse than no eval because it trains the team to ignore red. And run them concurrently for speed but never against shared state, which is the same rule written twice.
What to record per run#
The eval output should be a row per run, not a single number, because the aggregate hides everything you would want to act on. Keep the task id, whether it passed, turns, cost, wall-clock time, the stop reason, and the sequence of tool names. The last of those is close to free and pays for itself the first time a task starts failing: comparing the tool sequence of a failing run with the sequence from when it passed usually locates the change without reading any prose at all.
- Pass rate, with the failing task ids listed rather than counted.
- Mean and worst-case turns and cost. The mean hides the one run that took thirty turns.
- How many runs stopped on a budget rather than because the model finished.
- Tool error rate: how many tool calls returned an error, which usually moves before the pass rate does.
- Run the whole set more than once. Agents are non-deterministic, so a pass rate from a single pass over twenty tasks has a margin of several points on it.
Where people go wrong
Only evaluating the happy path. Agents fail on the strange cases, so an eval set of clean inputs measures the scenario you were never worried about.
If this comes up in an interview
How do you evaluate an agent rather than a single completion?
Check the end state of the world rather than the text, because the agent's job is to do something. Then track efficiency alongside correctness: turns and cost per task. A run that succeeds after fourteen turns is not equivalent to one that succeeds in three.
Your agent's success rate is unchanged after a prompt edit. Is it safe to ship?
Not necessarily. Check turns and cost as well. A change can hold the pass rate while doubling spend, and nothing in a pass-rate-only eval will tell you.
How many eval tasks do you need?
Twenty real ones beats two hundred synthetic ones. Draw them from actual usage, include every failure you have already seen in production, and version the set so a score from last month still means something.
Build this: Build a ten-task eval for an agent that reports pass rate, mean turns and mean cost in one command. Then make a change you believe is an improvement and see whether all three numbers agree with you.
Watch it explained
This is one section of The agent loop, 14 hours across 4 sections. Read it in full to get the whole topic in order.
Stop 16 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.