The agent loop
A while loop with tool dispatch and a stop condition.
Agent is a loaded word for something small. Call the model; if the reply asks to use a tool, run the tool, add the result to the conversation, and call the model again. Keep going until it stops asking or until you decide it has had enough turns. That loop, a few dozen lines of ordinary code, is the agent — there is no other machinery hiding behind the word.
One call to a model can only take one step. A loop lets it break a task into steps, see what actually happened after each one, and change course when something fails. What the loop does not have is judgement about stopping: without a cap on turns, a ceiling on spend and some check for repeated actions, a stuck agent will call the same failing tool hundreds of times and bill you for every one.
It is built on tool calling and on almost nothing else, which is why writing one by hand teaches more than any framework. Everything above it addresses one of its weaknesses: context management for when the conversation outgrows the window, agent reliability for when steps fail or repeat, multi-agent for splitting the work up, and MCP for where the tools come from.
An agent is a loop around a model call that can use tools, and the hard part is deciding when it stops.
In more detailCall the model; if it requested tools, run them and append the results; repeat until it stops or you hit a budget. That is genuinely all an agent is, and writing one by hand in under 300 lines teaches more than any framework. The difficulty is the guards: turn budgets, cost ceilings, loop detection, and knowing when to give up.
Learn first
What an agent actually is
By the end: You can draw the agent loop and explain every step of it.
- The loop, in five lines
- Why it lives in your code
- Writing one without a framework
- When an agent is the wrong answer
The word 'agent' is used loosely enough that people arrive expecting something exotic. This section takes it apart and builds one from nothing, so that you can see which part is the model and which part is your own code, a distinction that decides where every bug you meet later actually lives. It also covers the decision that comes first: whether the task in front of you needs an agent at all, which is where most projects go wrong.
A loop, not a mystery#
An agent is a while loop around a model call. Call the model. If it asked for tools, run them and add the results. Call it again. Stop when it stops asking or when you run out of budget.
1. Send messages and tool definitions
2. Did the model request a tool?
no -> done, return its answer
yes -> run the tools, append the results, go to 1That is the whole idea. Everything else in agent engineering is about what you do around that loop: budgets, error handling, deciding what stays in the context.
Write it yourself once#
def run(task: str, max_turns: int = 10) -> str:
messages = [{"role": "user", "content": task}]
for turn in range(max_turns):
response = client.chat(messages, tools=TOOLS)
messages.append(response.message)
if response.stop_reason != "tool_use":
return response.text # model is done
for call in response.tool_calls:
result = run_tool(call)
messages.append({
"role": "tool",
"tool_call_id": call.id,
"content": json.dumps(result),
})
return "Stopped: reached the turn limit."Why write it by hand
Interviewers ask you to reconstruct this loop and defend the decisions in it. Candidates who have only used a framework can describe what it does and not why. Twenty minutes writing it yourself is the difference.
The same loop, actually running#
Reading a loop and watching one are different. Everything in this topic runs against a scripted stand-in for the model rather than a hosted one, so the loop, the failures in it and the guards against them all execute offline and print the same numbers on every run. The stub takes a list of responses and returns them in order, which is exactly what a real provider does, minus the uncertainty about which response you will get.
import json
def find_order(customer):
return {"order_id": "A-4471", "total_cents": 2900}
def refund(order_id):
return {"refunded": order_id}
TOOLS = {"find_order": find_order, "refund": refund}
class StubModel:
"""A scripted stand-in for a hosted model, so the loop runs offline and
prints the same thing on every run. One script entry per response."""
def __init__(self, script):
self.script, self.calls = list(script), 0
def chat(self, messages, tools=None):
self.calls += 1
return self.script[min(self.calls, len(self.script)) - 1]
def run(model, task, max_turns=6):
messages = [{"role": "user", "content": task}]
transcript = []
for turn in range(max_turns):
reply = model.chat(messages, tools=TOOLS)
messages.append({"role": "assistant", "content": reply})
if reply["stop_reason"] != "tool_use":
return reply["text"], transcript
for call in reply["tool_calls"]:
result = TOOLS[call["name"]](**call["args"])
transcript.append((turn, call["name"], call["args"], result))
messages.append({"role": "tool", "content": json.dumps(result)})
return "stopped: reached the turn limit", transcript
SCRIPT = [
{"stop_reason": "tool_use",
"tool_calls": [{"name": "find_order", "args": {"customer": "ada"}}]},
{"stop_reason": "tool_use",
"tool_calls": [{"name": "refund", "args": {"order_id": "A-4471"}}]},
{"stop_reason": "end_turn", "text": "Refunded order A-4471."},
]
answer, transcript = run(StubModel(SCRIPT), "Refund Ada's most recent order.")
for turn, name, args, result in transcript:
print(f"turn {turn}: {name}({args}) -> {result}")
print("answer:", answer)
# turn 0: find_order({'customer': 'ada'}) -> {'order_id': 'A-4471', 'total_cents': 2900}
# turn 1: refund({'order_id': 'A-4471'}) -> {'refunded': 'A-4471'}
# answer: Refunded order A-4471.Every tool call is untrusted input#
The loop above indexes straight into the tool table and splats the arguments in. That works for exactly as long as the model emits the call you imagined. It will not: it will misname a key, drop a required argument, or ask for a tool that does not exist, usually because something similar existed in its training data. None of those are exotic and none of them should end the run.
BAD_CALLS = [
{"name": "refund", "args": {"order_id": "A-4471"}}, # fine
{"name": "refund", "args": {"order": "A-4471"}}, # wrong key
{"name": "refund", "args": {}}, # nothing at all
{"name": "refund_ord", "args": {"order_id": "A-4471"}}, # invented tool
]
def dispatch_naive(call):
return TOOLS[call["name"]](**call["args"])
def dispatch(call):
"""Never raises. Every outcome is something the model can read and act on."""
fn = TOOLS.get(call["name"])
if fn is None:
return {"ok": False, "error": "no tool named " + call["name"]
+ ". Available: " + ", ".join(sorted(TOOLS))}
try:
return {"ok": True, "result": fn(**call["args"])}
except TypeError as e:
return {"ok": False, "error": "bad arguments: " + str(e)}
except Exception as e:
return {"ok": False, "error": type(e).__name__ + ": " + str(e)}
survived = 0
for call in BAD_CALLS:
try:
dispatch_naive(call)
survived += 1
except Exception as e:
print("naive loop died:", type(e).__name__)
print("naive dispatcher survived", survived, "of", len(BAD_CALLS), "calls")
print()
for call in BAD_CALLS:
print(dispatch(call))
# naive loop died: TypeError
# naive loop died: TypeError
# naive loop died: KeyError
# naive dispatcher survived 1 of 4 calls
#
# {'ok': True, 'result': {'refunded': 'A-4471'}}
# {'ok': False, 'error': "bad arguments: refund() got an unexpected keyword argument 'order'"}
# {'ok': False, 'error': "bad arguments: refund() missing 1 required positional argument: 'order_id'"}
# {'ok': False, 'error': 'no tool named refund_ord. Available: find_order, refund'}That is the rule for errors inside a tool generally, not just argument errors. A database timeout, a 404, a rate limit: catch it, put the reason in the tool result, and let the model decide what to do. The one distinction worth making is between an error the model can act on and one it cannot. A missing argument is actionable. A payment provider being down is not, and telling the model about it just invites it to try again with the same arguments, which is why that class of error belongs to your retry policy rather than to the model.
Designing a tool the model can call correctly#
You cannot stop a model emitting a bad call, but you can decide what a bad call does. Most of that is decided by the tool signature, before the model is involved. Three rules cover most of it: no default that silently acts, the unit in the parameter name, and closed sets instead of free strings. The demonstration is a partial refund, where the customer is owed 9.00 on an order worth 29.00.
ORDERS = {"A-4471": {"total_cents": 2900, "status": "shipped"}}
WANT = 900 # the customer is owed 9.00, not the whole order
def refund_loose(order_id, amount_cents=None):
"""Defaults that act. Omitting the amount refunds everything."""
amount = ORDERS[order_id]["total_cents"] if amount_cents is None else amount_cents
return {"ok": True, "refunded_cents": amount}
def refund_strict(order_id, amount_cents):
"""No default, one unit in the name, and the value checked against the order."""
if not isinstance(amount_cents, int):
return {"ok": False, "error": "amount_cents must be a whole number of cents, "
"got " + repr(amount_cents)}
total = ORDERS[order_id]["total_cents"]
if not 0 < amount_cents <= total:
return {"ok": False, "error": f"amount_cents must be between 1 and {total}"}
return {"ok": True, "refunded_cents": amount_cents}
EMITTED = [
{"order_id": "A-4471", "amount_cents": 900}, # what you hoped for
{"order_id": "A-4471"}, # forgot the amount
{"order_id": "A-4471", "amount_cents": 9.0}, # dollars, not cents
]
for label, fn in (("loose", refund_loose), ("strict", refund_strict)):
for args in EMITTED:
try:
out = fn(**args)
except TypeError as e:
out = {"ok": False, "error": "bad arguments: " + str(e)}
verdict = ("correct" if out.get("refunded_cents") == WANT
else "REJECTED" if not out["ok"]
else "SILENTLY WRONG: refunded " + str(out["refunded_cents"]))
print(f"{label:<7}{str(args):<50}{verdict}")
# loose {'order_id': 'A-4471', 'amount_cents': 900} correct
# loose {'order_id': 'A-4471'} SILENTLY WRONG: refunded 2900
# loose {'order_id': 'A-4471', 'amount_cents': 9.0} SILENTLY WRONG: refunded 9.0
# strict {'order_id': 'A-4471', 'amount_cents': 900} correct
# strict {'order_id': 'A-4471'} REJECTED
# strict {'order_id': 'A-4471', 'amount_cents': 9.0} REJECTEDNotice what the strict version did not do: it did not make the model more accurate. Both signatures received the same three calls. What changed is that a wrong call became visible and correctable instead of a silent refund of the wrong amount. That is the whole of tool schema design.
- One tool, one job. A do_everything tool with a mode argument moves the decision into free text, where you cannot validate it.
- Name the unit and the type in the parameter: amount_cents, not amount. Days, not duration.
- No default that acts. A default that reads is fine; a default that refunds, deletes or sends is not.
- Closed sets over free strings. status must be one of shipped, refunded, cancelled, and the error should list them.
- Return little. Whatever a tool returns is resent on every later turn, which is the context problem in the section after next.
- Describe the tool for someone who has not seen your codebase, and say when not to use it. The description is prompt, and it is read every turn.
When not to build an agent#
Agents are slow, expensive and harder to debug than a single call. They earn that cost only when the steps genuinely cannot be decided in advance.
- One model call: classification, extraction, summarising, rewriting.
- A fixed pipeline you control: retrieve, then answer. You know the steps, so hard-code them.
- An agent: the number and order of steps depend on what is found along the way.
The middle row hides the interesting case, which is a pipeline with a decision in it. Branching on the result of a step does not make something an agent. The difference is who enumerated the paths: in a pipeline you did, in advance, and there are as many as you wrote; in an agent the model does, at run time, and there is no upper bound on them except the one you impose. Costing that out on the same set of tasks makes the trade concrete.
# Each task needs some number of lookups before the answer is knowable. You do
# not know that number in advance; that is the whole difficulty.
DEPTHS = [1, 1, 1, 2, 1, 3, 8]
MAX_CALLS = 6
def pipeline(depth):
"""Look up once, then answer. Two model calls, always."""
return 2, depth <= 1
def pipeline_with_a_decision(depth):
"""Look up; if that missed, look up in the other system; then answer."""
return min(depth, 2) + 1, depth <= 2
def agent(depth):
"""Keep going until the model stops asking, or the cap trips."""
calls = min(depth + 1, MAX_CALLS)
return calls, depth + 1 <= MAX_CALLS
print(f"{'strategy':<26}{'model calls':>12}{'solved':>9}")
for name, fn in (("pipeline", pipeline),
("pipeline with a decision", pipeline_with_a_decision),
("agent", agent)):
runs = [fn(d) for d in DEPTHS]
calls = sum(c for c, _ in runs)
solved = sum(ok for _, ok in runs)
print(f"{name:<26}{calls:>12}{solved:>6}/{len(DEPTHS)}")
print("worst case calls: pipeline 2, decision 3, agent", MAX_CALLS, "(the cap)")
# strategy model calls solved
# pipeline 14 4/7
# pipeline with a decision 17 5/7
# agent 21 6/7
# worst case calls: pipeline 2, decision 3, agent 6 (the cap)Read the last two columns together rather than separately. The agent solves the most tasks and it is the only one of the three whose cost you cannot state before you run it, which is why its worst case is a number you chose rather than a number the design gives you. It also fails the deepest task after spending six calls, where the pipeline fails it after two. That asymmetry is the honest case against agents: you pay the full price on the runs that do not work as well as the ones that do.
So the question to ask of a task is not whether it is complicated but whether you can write the paths down. If you can, write them down; you get a cheaper system with a knowable worst case and a stack trace when it breaks. Saying this out loud in an interview is a strong signal. Reaching for an agent when a pipeline would do is the most common design mistake in this area.
Where people go wrong
Building an agent because the task sounds sophisticated. If you can write down the steps in advance, write them down in code, and you will get something faster, cheaper and testable.
Build this: Write the loop above from scratch with one tool and a turn limit. No framework. Then give it a task it cannot finish and confirm it stops cleanly.
Open this section on its own pageBudgets, loops and giving up well
By the end: Your agent cannot run forever or spend unbounded money, and it explains why it stopped.
- Turn, time and cost limits
- Detecting a stuck agent
- Returning a partial result instead of raising
- Logging enough to reconstruct a run
An agent decides for itself how many steps to take, so nothing in the loop you have just written prevents it running for hours. This section puts ceilings on a run, on turns, on time and on money, and covers what should happen when one is reached. Stopping well matters as much as stopping: a run that returns what it managed and says why it gave up is still useful to whoever asked, and an exception is not.
Three limits, not one#
A turn limit alone is not enough. A single turn can be slow, and a small number of turns with large contexts can be expensive. Track all three.
import time
class Budget:
def __init__(self, max_turns=10, max_seconds=120, max_usd=0.50):
self.max_turns = max_turns
self.deadline = time.time() + max_seconds
self.max_usd = max_usd
self.turns = 0
self.spent = 0.0
def exceeded(self) -> str | None:
if self.turns >= self.max_turns: return "turn limit"
if time.time() > self.deadline: return "time limit"
if self.spent >= self.max_usd: return "cost limit"
return NoneEveryone's first agent incident
An agent loops overnight, calling the same tool with the same arguments several hundred times, and the bill arrives before anyone notices. Add budgets before the first deployment, not after the first invoice.
What an unbounded run actually costs#
The reason a stuck run is expensive rather than merely slow is that the cost of a turn is not constant. Every turn resends the whole conversation, so the input grows on each one and the bill grows with the square of the turn count. Doubling the turns roughly quadruples the spend. The numbers below come from running the same stuck agent four times with different guards, priced at three dollars per million input tokens and fifteen per million output.
IN_PER_M, OUT_PER_M = 3.00, 15.00 # dollars per million tokens
def turn_cost(turn):
"""Every turn resends the whole conversation, so input grows each turn."""
tokens_in = 1200 + 700 * turn # system and tools, plus what has piled up
tokens_out = 120
return tokens_in / 1e6 * IN_PER_M + tokens_out / 1e6 * OUT_PER_M
class Budget:
def __init__(self, max_turns=10, max_usd=0.50):
self.max_turns, self.max_usd = max_turns, max_usd
self.turns, self.spent = 0, 0.0
def charge(self, turn):
self.spent += turn_cost(turn)
self.turns += 1
def exceeded(self):
if self.turns >= self.max_turns: return "turn limit"
if self.spent >= self.max_usd: return "cost limit"
return None
STUCK = [{"stop_reason": "tool_use",
"tool_calls": [{"name": "search", "args": {"q": "refund policy"}}]}]
class StubModel:
def __init__(self, script):
self.script, self.calls = list(script), 0
def chat(self, messages, tools=None):
self.calls += 1
return self.script[min(self.calls, len(self.script)) - 1]
def run_bounded(model, budget, detect_repeats=False, hard_cap=200):
last_sig, repeats = None, 0
for turn in range(hard_cap):
why = budget.exceeded()
if why:
return turn, why, budget.spent
reply = model.chat([])
budget.charge(turn)
if reply["stop_reason"] != "tool_use":
return turn + 1, "finished", budget.spent
call = reply["tool_calls"][0]
sig = (call["name"], tuple(sorted(call["args"].items())))
repeats = repeats + 1 if sig == last_sig else 0
last_sig = sig
if detect_repeats and repeats >= 2:
return turn + 1, "repeat detected", budget.spent
return hard_cap, "hard cap", budget.spent
for label, budget, detect in (
("no bounds at all", Budget(max_turns=200, max_usd=1e9), False),
("turn limit 10", Budget(max_turns=10, max_usd=1e9), False),
("cost limit $0.15", Budget(max_turns=200, max_usd=0.15), False),
("repeat detector", Budget(max_turns=200, max_usd=1e9), True),
):
turns, why, spent = run_bounded(StubModel(STUCK), budget, detect)
print(f"{label:<18}{turns:>4} turns ${spent:>6.3f} stopped on {why}")
# no bounds at all 200 turns $42.870 stopped on hard cap
# turn limit 10 10 turns $ 0.148 stopped on turn limit
# cost limit $0.15 11 turns $ 0.175 stopped on cost limit
# repeat detector 3 turns $ 0.022 stopped on repeat detectedTwo details in that table repay a second look. The cost limit stopped at 0.175 rather than 0.15, because a budget is checked after the turn that breached it and the last turn is always the most expensive one; if the overshoot matters, estimate the next turn from the current context size and refuse to start it. And the repeat detector cost less than a hundredth of the unbounded run, which is the general shape of this: budgets stop a disaster becoming a catastrophe, and noticing why the agent is stuck stops it being a disaster.
Spotting a stuck agent#
The classic failure is calling the same tool with the same arguments repeatedly, because the result did not help and the model has no other idea. Detect it and intervene.
import json
def repeat_guard(call, last_signature, repeats):
"""Returns the tool result to send back, or None to run the tool."""
signature = (call["name"], json.dumps(call["args"], sort_keys=True))
if signature == last_signature:
repeats += 1
if repeats >= 2:
return signature, repeats, {
"ok": False,
"error": "You have called this tool with identical arguments "
"three times. Try a different approach or stop."}
else:
repeats = 0
return signature, repeats, None
CALLS = [{"name": "search", "args": {"q": "refund policy"}}] * 4
last, repeats = None, 0
for i, call in enumerate(CALLS):
last, repeats, intervention = repeat_guard(call, last, repeats)
print(i, "repeats =", repeats, "->", intervention or "run the tool")
# 0 repeats = 0 -> run the tool
# 1 repeats = 1 -> run the tool
# 2 repeats = 2 -> {'ok': False, 'error': 'You have called this tool with identical arguments three times. Try a different approach or stop.'}
# 3 repeats = 3 -> {'ok': False, 'error': 'You have called this tool with identical arguments three times. Try a different approach or stop.'}Exact argument equality is the cheap version and it misses the common case, which is a loop that is stuck without repeating itself precisely: the same two tools alternating, or the same search with the query reworded each time. Two upgrades are worth the lines. Compare a canonical form of the arguments, lowercased with whitespace collapsed, so a reworded query still matches. And keep the last few signatures rather than one, so an A-B-A-B cycle is caught as well as A-A-A. The agent-reliability topic measures how much each of those catches on a mixture of stuck runs, because the answer is not obvious and it decides how much of this is worth building.
Whatever you detect, prefer telling the model over killing the run. A tool result saying it has now called this three times with the same arguments and should try something else costs one turn and often works, and if it does not, you still have the turn limit underneath.
The stop condition that never fires#
The other way a loop runs forever is subtler, and it is your bug rather than the model's. The loop stops when a condition is true, and the condition is written against the text instead of against the structure. A model told to end with the word DONE ends with the word DONE most of the time, which is the worst possible hit rate for a stop condition, because it works in testing.
FINALS = [
"DONE. The order has been refunded.",
"The order has been refunded.",
"I have refunded the order. Let me know if you need anything else.",
"Refund complete - task done.",
"All finished!",
]
def stops_on_text(text):
return "DONE" in text
def stops_on_stop_reason(reply):
return reply["stop_reason"] != "tool_use"
replies = [{"stop_reason": "end_turn", "text": t} for t in FINALS]
print("string check :", sum(stops_on_text(r["text"]) for r in replies), "/", len(replies))
print("stop_reason :", sum(stops_on_stop_reason(r) for r in replies), "/", len(replies))
# string check : 1 / 5
# stop_reason : 5 / 5Stop on the structured field the provider gives you, which the model cannot phrase differently. If the task genuinely needs a completion signal that the field cannot express, make it a tool the model calls, such as a finish tool taking the answer as an argument, so that finishing is a call you can validate rather than a phrase you have to match. The same argument applies anywhere you are tempted to parse intent out of prose the model wrote.
Stop usefully#
When a budget runs out, do not raise. Return what was achieved, what remains, and why it stopped. A partial answer with an explanation is useful to a person; an exception is not.
from dataclasses import dataclass
@dataclass
class AgentResult:
answer: str
complete: bool
stopped_because: str
turns_used: int
cost_usd: float
def give_up(best_so_far, why, turns_used, cost_usd):
return AgentResult(
answer=best_so_far,
complete=False,
stopped_because=why,
turns_used=turns_used,
cost_usd=cost_usd,
)
print(give_up("Found the order but could not confirm the refund.",
"cost limit", turns_used=11, cost_usd=0.175))
# AgentResult(answer='Found the order but could not confirm the refund.', complete=False, stopped_because='cost limit', turns_used=11, cost_usd=0.175)Log every turn#
Without a record of each turn you are guessing when something goes wrong. Log the tools requested, the arguments, the result size, the tokens and the cumulative cost. This is the beginning of tracing, and the first thing anyone asks about a misbehaving agent is what it actually did.
It is also, almost always, how you debug one. There is no debugger for a loop that did the wrong thing, because nothing went wrong in the ordinary sense: every tool returned successfully and the code did what it was told. What you have is a sequence of decisions, and the question is which one was the first bad one. So the log is not an operations nicety, it is the debugger, and it is worth designing as one.
import json
# One real run, logged. The customer asked about order 4471 and got a refund
# on a different order entirely.
LOG = [
{"turn": 0, "tool": "search_customer", "args": {"name": "bob"},
"result": {"matches": ["C-118", "C-402"]}, "tokens_in": 1200},
{"turn": 1, "tool": "list_orders", "args": {"customer": "C-402"},
"result": {"orders": ["A-9002"]}, "tokens_in": 1850},
{"turn": 2, "tool": "refund", "args": {"order_id": "A-9002", "amount_cents": 4500},
"result": {"ok": True}, "tokens_in": 2600},
]
def show(log):
print(f"{'turn':>4} {'tool':<16}{'arguments':<46}{'tokens_in':>9}")
for e in log:
print(f"{e['turn']:>4} {e['tool']:<16}{json.dumps(e['args']):<46}{e['tokens_in']:>9}")
def first_appearance(log, value):
"""Where did the agent get this from? The single most useful query."""
for e in log:
if value in json.dumps(e["result"]):
return e["turn"], e["tool"], e["args"]
return None
show(LOG)
print()
print("the refund used order", LOG[-1]["args"]["order_id"])
print("first seen at turn", first_appearance(LOG, "A-9002"))
print("that came from customer C-402, and turn 0 returned two matches:",
LOG[0]["result"]["matches"])
# turn tool arguments tokens_in
# 0 search_customer {"name": "bob"} 1200
# 1 list_orders {"customer": "C-402"} 1850
# 2 refund {"order_id": "A-9002", "amount_cents": 4500} 2600
#
# the refund used order A-9002
# first seen at turn (1, 'list_orders', {'customer': 'C-402'})
# that came from customer C-402, and turn 0 returned two matches: ['C-118', 'C-402']The bug is now obvious and it is not in turn two. Turn zero returned two customers called Bob and the model picked one, and nothing in the loop noticed that a lookup meant to identify one person returned two. The fix is in the tool, not the prompt: a search that matches more than one record should return an error saying so rather than a list, because a list invites a guess. That is the usual shape of an agent bug. The turn where the damage happened is rarely the turn where the mistake was made.
- Log the arguments in full, not a summary. The wrong argument is the bug most of the time.
- Log the result, truncated with the original length recorded, so you can tell a tool returning nothing from a tool returning ten thousand tokens.
- Log tokens in, tokens out and cumulative cost per turn, or you cannot answer why a run cost what it did.
- Give every run an id and put it in every log line, because you will be reading two interleaved runs sooner than you think.
- Keep the exact system prompt and tool definitions with the run. A run you cannot reproduce because the prompt has moved on is an anecdote.
One habit is worth more than all of them: before changing anything, read the transcript to the end and say out loud what you think the agent believed at each turn. Most agent bugs that survive an afternoon are cases where the engineer fixed the turn that looked wrong instead of the turn that made it inevitable.
Where people go wrong
Only limiting turns. A three-turn run with a very large context can cost more than a twenty-turn run with small ones, so a turn limit gives you no real protection against spend.
Build this: Add all three budgets to your loop, then give the agent an impossible task and confirm it stops on each limit in turn by tightening one at a time.
Open this section on its own pageManaging the context as it grows
By the end: Your agent can run for many turns without overflowing or becoming unaffordable.
- Why context grows faster than you expect
- Trimming, summarising and offloading
- Keeping the system prompt stable for caching
- What must never be dropped
Every turn an agent takes adds to the pile of text resent on the next one, so a run that starts cheap becomes expensive and eventually will not fit at all. This section is about deciding what stays and what goes: trimming tool results, summarising old turns, moving bulk out to storage, and ordering the prompt so none of that destroys your caching. It also covers what must survive any amount of compaction, because an agent that has lost sight of what it was asked does not stop, it carries on doing something else.
The growth problem#
Every turn adds the model's message and every tool result, and all of it is resent next turn. Tool results are usually the bulk of it. An agent that reads three documents and searches twice can be carrying tens of thousands of tokens by turn six.
Turn 1: 2,400 tokens
Turn 3: 9,100 tokens
Turn 6: 31,000 tokens same task, nothing unusual happenedIt is worth seeing where that comes from rather than taking it on trust, because the answer decides which fix is worth building. The run below is twelve turns alternating a search and a document read, with the tools returning what tools return by default: the whole search response and the whole page. Token counts use characters over four, which is not your provider's tokeniser but is deterministic and close enough to reason about.
import json
def tokens(s):
"""Characters over four. Not your provider's tokeniser, but deterministic,
close enough to reason with, and it makes this block reproducible."""
return max(1, len(s) // 4)
PARA = ("Refunds on the annual plan are available within fourteen days of the "
"renewal date, and are returned to the original payment method within "
"five working days of approval. Partial refunds are calculated pro rata "
"from the cancellation date. ")
DOCUMENT = {"doc_id": "POL-14", "title": "Refunds and cancellations",
"text": PARA * 40} # a policy page, in full
SEARCH_HITS = [{"order_id": "A-44" + str(70 + i), "customer_id": "C-118",
"status": "shipped", "total_cents": 2900 + i,
"snippet": PARA[:220] + " (order A-44" + str(70 + i) + ")"}
for i in range(20)]
SCRIPT = [("search_orders", SEARCH_HITS), ("read_document", DOCUMENT)] * 6
SYSTEM = 900 # tokens of system prompt and tool definitions
ASSISTANT = 120 # the model's own message each turn
IN_PER_M = 3.00
def slim(name, result):
"""What the tool should have returned: the fields the model can act on."""
if name == "search_orders":
return [{"order_id": r["order_id"], "status": r["status"]} for r in result[:5]]
return {"doc_id": result["doc_id"], "title": result["title"],
"text": result["text"][:400] + "..."}
def run(policy, keep_recent=3):
context, peak, billed, store = [], 0, 0, {}
for turn, (name, result) in enumerate(SCRIPT):
if policy == "trim in the tool":
body = json.dumps(slim(name, result))
elif policy == "offload":
key = name + "-" + str(turn)
store[key] = result
body = json.dumps({"stored_as": key,
"rows": len(result) if isinstance(result, list) else 1})
else:
body = json.dumps(result)
context.append({"turn": turn, "name": name, "body": body})
if policy == "drop stale results":
for e in context[:-keep_recent]:
e["body"] = json.dumps({"note": e["name"] + " ran at turn "
+ str(e["turn"]) + ", result dropped"})
size = SYSTEM + sum(tokens(e["body"]) + ASSISTANT for e in context)
peak = max(peak, size)
billed += size
return peak, billed, billed / 1e6 * IN_PER_M
print("one search_orders result:", tokens(json.dumps(SEARCH_HITS)), "tokens ->",
tokens(json.dumps(slim("search_orders", SEARCH_HITS))), "trimmed")
print("one read_document result:", tokens(json.dumps(DOCUMENT)), "tokens ->",
tokens(json.dumps(slim("read_document", DOCUMENT))), "trimmed")
print()
print(f"{'policy':<22}{'peak context':>13}{'billed in':>11}{'cost':>9}")
for policy in ("as written", "drop stale results", "trim in the tool", "offload"):
peak, billed, cost = run(policy)
print(f"{policy:<22}{peak:>13,}{billed:>11,} ${cost:>6.3f}")
# one search_orders result: 1700 tokens -> 56 trimmed
# one read_document result: 2387 tokens -> 118 trimmed
#
# policy peak context billed in cost
# as written 26,862 177,492 $ 0.532
# drop stale results 8,931 87,837 $ 0.264
# trim in the tool 3,384 26,760 $ 0.080
# offload 2,467 20,983 $ 0.063The first two lines are the whole argument. A search returning twenty hits with a snippet each is 1,700 tokens, and the five order ids the model will actually act on are 56. You are paying thirty times over for fields nobody reads, and paying it again on every subsequent turn. That is why the ordered list below starts where it does.
Four things to do about it#
- 1.Return less in the first place. Summarise or truncate inside the tool, before the result ever enters the context. Cheapest fix by far.
- 2.Drop old tool results. A search result from turn two is rarely needed at turn nine. Replace it with a one-line note that it happened.
- 3.Summarise older turns into a running summary, keeping recent turns verbatim.
- 4.Offload to storage. Write the full result to a file or database and keep only an identifier the model can use to fetch it again.
def compact(messages, keep_recent=6):
if len(messages) <= keep_recent + 1:
return messages
system, old, recent = messages[0], messages[1:-keep_recent], messages[-keep_recent:]
summary = client.chat([
{"role": "user",
"content": "Summarise this agent transcript. Keep every decision, "
"finding and open question. Drop the raw tool output.\n\n"
+ render(old)},
]).text
return [system, {"role": "user", "content": f"Earlier:\n{summary}"}, *recent]Keep the front of the prompt stable#
Caching is prefix-matched
Providers cache the processed form of a stable prompt prefix, which makes repeated calls much cheaper. It matches from the start, so one changed byte early invalidates everything after it. Put a timestamp in your system prompt and you have silently switched off your largest cost saving.
Order everything from most stable to most volatile:
system prompt never changes
tool definitions rarely change
few-shot examples rarely change
summarised history changes slowly
recent turns changes every turn
current question changes every turnAgent loops are the best case for a cache, because the message list only ever grows at the end. Everything sent on turn five was sent on turn four in exactly the same order, so on a well-behaved loop the entire previous context is a cache hit and only the newest turn is fresh. Pricing that out on the same twelve-turn run says how much the discipline is worth, and what one careless line costs.
# Assumed prices: a cache read costs a tenth of a fresh input token, and
# writing to the cache costs a quarter more. Check your provider's numbers.
READ, WRITE = 0.10, 1.25
def sizes(policy, keep_recent=3):
"""Context size at each turn, using the same machinery as above."""
out, context = [], []
for turn, (name, result) in enumerate(SCRIPT):
body = json.dumps(slim(name, result) if policy == "trim in the tool" else result)
context.append({"turn": turn, "name": name, "body": body})
out.append(SYSTEM + sum(tokens(e["body"]) + ASSISTANT for e in context))
return out
def cost(policy, stable_prefix=True, compact_every=None):
"""Append-only messages mean everything before this turn is a cache hit."""
sz, total, cached = sizes(policy), 0.0, 0
for turn, size in enumerate(sz):
if not stable_prefix: # a timestamp in the system prompt
cached = 0
if compact_every and turn and turn % compact_every == 0:
cached = 0 # the middle was rewritten
fresh = size - cached
total += (cached * READ + fresh * WRITE) / 1e6 * IN_PER_M
cached = size
return total
for policy in ("as written", "trim in the tool"):
plain = run(policy)[2]
print(f"{policy:<18} no cache ${plain:.3f} cached ${cost(policy):.3f} "
f"cached but with a timestamp in the system prompt "
f"${cost(policy, stable_prefix=False):.3f}")
print("compacting every 4 turns, as written: $%.3f" % cost("as written", compact_every=4))
# as written no cache $0.532 cached $0.146 cached but with a timestamp in the system prompt $0.666
# trim in the tool no cache $0.080 cached $0.020 cached but with a timestamp in the system prompt $0.100
# compacting every 4 turns, as written: $0.242That third column is the line to remember. A cache you keep invalidating is not neutral, it is a 25% surcharge, because you pay the write premium on every turn and never collect a read. The same is true of anything else that moves at the front: a session id, a list of tools ordered by a dictionary that does not preserve order, a few-shot example chosen at random per call.
The last line is the tension nobody mentions. Compaction rewrites the middle of the message list, so the turn after a compaction is a complete cache miss. Compacting every four turns on this run costs 0.242 against 0.146 for leaving it alone, which means on a twelve-turn run compaction lost money. It starts winning when the context is large enough that the saved tokens outweigh the lost cache hits, or when you were going to overflow the window regardless. Compact on a size threshold, not on a turn count, and measure whether it paid.
What not to drop#
- The original task. Agents that lose sight of what they were asked wander.
- Anything the user explicitly said, especially constraints and corrections.
- Decisions already made, so the agent does not reverse them.
- Failures already encountered, or it will retry them.
That list is easy to agree with and hard to implement, because a summariser does not know which of the things it is reading will matter in four turns. The honest way to see the problem is to write down what a later turn will need, compact, and then check. Below, the summariser is a stand-in that keeps the gist and the last three turns; real ones drop different things, and they all drop something.
TURNS = [
"User: refund my last order, but to the original card only, not store credit.",
"search_orders(customer=C-118) -> 3 orders, most recent A-4471",
"get_order(A-4471) -> total 2900 cents, status shipped",
"refund(A-4471, method=store_credit) -> ERROR: customer opted out of credit",
"User: I said the card. Also the address on file is wrong, use 14 Bridge Street.",
"get_customer(C-118) -> card ending 4242, address 9 Mill Road",
"update_address(C-118, '14 Bridge Street') -> ok",
"search_policy('partial refund') -> pro rata from the cancellation date",
]
# The facts a later turn will need. Written out so the check is honest about
# what it is testing rather than grading a summary on vibes.
MUST_SURVIVE = {
"original card, not store credit": "original card",
"the order is A-4471": "A-4471",
"store credit already failed": "opted out",
"address corrected to Bridge St": "Bridge Street",
}
def naive_summary(turns, keep=3):
"""A stand-in for a summarising model that keeps the gist and the recent
turns. Real summarisers drop different things; they all drop something."""
return "Earlier: the user asked about a refund. " + " ".join(turns[-keep:])
def ledger(turns):
"""Facts are extracted as they appear and carried verbatim, not summarised."""
facts = [t for t in turns if any(m in t for m in MUST_SURVIVE.values())]
return "Known facts:\n" + "\n".join("- " + f for f in facts)
for name, text in (("naive summary", naive_summary(TURNS)), ("fact ledger", ledger(TURNS))):
kept = [label for label, needle in MUST_SURVIVE.items() if needle in text]
lost = [label for label in MUST_SURVIVE if label not in kept]
print(f"{name:<15}{len(kept)}/{len(MUST_SURVIVE)} survived lost: {lost}")
print(f"{'':<15}{len(text) // 4} tokens")
print("the eight turns in full:", len("\n".join(TURNS)) // 4, "tokens")
# naive summary 1/4 survived lost: ['original card, not store credit', 'the order is A-4471', 'store credit already failed']
# 54 tokens
# fact ledger 4/4 survived lost: []
# 105 tokens
# the eight turns in full: 131 tokensRead the lost list rather than the score. It dropped the constraint the user repeated after the agent got it wrong, and it dropped the record of that failure, which between them guarantee the agent tries store credit again. That is the specific way compacted agents go into a loop: compaction removes the memory of the thing that failed, so the same thing is tried, fails, and is compacted away once more.
So do not rely on a summary to carry the facts. Keep a small structured ledger alongside the messages, appended to as turns happen: the task as stated, every constraint the user gave, every decision taken, every tool call that failed and why. It is a few hundred tokens, it never gets summarised, and it goes in the prompt after the stable prefix. The summary is then free to be lossy, because nothing depends on it.
Where people go wrong
Summarising with the same expensive model you are running the agent with. Compaction can quietly cost more than the work. Use a cheaper model for summarising, and measure it.
Build this: Run an agent until it overflows, then add compaction and measure three things before and after: total tokens, total cost, and whether the task still succeeds. The third is the one people forget.
Open this section on its own pageKnowing whether it works
By the end: You have a number that tells you whether a change to the agent helped.
- Defining success when the path matters
- A fixed task set run on every change
- Tracking cost and turns alongside success
- The regression that hides behind a passing score
You can build an agent and stop it running away. What you cannot yet do is say whether last week's prompt edit made it better. This section is about getting a number: what success means when the job is to change something in the world rather than produce text, and why turns and cost belong next to the pass rate. Without it, every later change to the agent is a guess nobody can check.
Watching is not measuring#
Agents are compelling to watch. You see it reason, call a tool, recover, and conclude, and it feels like evidence. It is not. You are seeing one run of a non-deterministic system, chosen by you, at a moment when you were hoping it would work.
This is where most agent projects stall. The demo is impressive, nobody can say whether last week's change helped, and eventually nobody wants to touch the prompt.
Defining success#
For an agent, the final answer is not the only thing that matters. A run that reaches the right answer after fourteen turns and three wrong tool calls is not a success in any useful sense.
- Outcome: is the end state correct? Was the record actually updated, the file actually written?
- Efficiency: how many turns and how much money did it take?
- Path: did it use sensible tools, or stumble into the answer?
Outcome and efficiency are easy to write down. Path is the one people either skip or overdo. The overdone version pins an exact sequence of tool calls and breaks the moment the agent finds a better route, which teaches you nothing except that it changed. The useful version is weaker and survives: assert that the required tool was called at all, that no forbidden tool was called, and that nothing was called more than once that should have been called once. Those three catch the cases you care about, and a run that reaches the right answer by a route you did not anticipate still passes.
from __future__ import annotations
from dataclasses import dataclass
from typing import Callable
@dataclass
class Task:
prompt: str
check: Callable[[AgentResult], bool]
max_turns: int = 8
max_usd: float = 0.20
TASKS = [
Task(
prompt="Refund order 4471 and email the customer.",
check=lambda r: order(4471).status == "refunded" and emails_sent() == 1,
),
...
]Check side effects, not wording
Asserting on the agent's closing sentence is fragile and misses the point. The question is whether the refund happened and whether exactly one email went out. Check the world.
Run it on every change#
import asyncio
from statistics import mean
async def evaluate(tasks):
results = await asyncio.gather(*(run_one(t) for t in tasks))
print(f"passed: {sum(r.passed for r in results)}/{len(results)}")
print(f"avg turns: {mean(r.turns for r in results):.1f}")
print(f"avg cost: ${mean(r.cost for r in results):.3f}")
print(f"total: ${sum(r.cost for r in results):.2f}")Twenty tasks is enough to be useful. Include the failures you have already seen in production, because those are the cases most likely to come back.
A harness, small enough to read#
The pieces are a world you can reset, tools that change it, tasks that assert on it afterwards, and three versions of the same agent to compare. Each version is written out as the calls it makes, so the comparison runs offline and reports the same numbers every time.
import copy
WORLD0 = {"orders": {"A-1": "shipped", "A-2": "shipped", "A-3": "shipped"},
"emails": [], "tickets": {"T-1": "open", "T-2": "open", "T-3": "open"}}
def refund(w, order_id): w["orders"][order_id] = "refunded"; return {"ok": True}
def send_email(w, to): w["emails"].append(to); return {"ok": True}
def close_ticket(w, tid): w["tickets"][tid] = "closed"; return {"ok": True}
def get_order(w, order_id): return {"status": w["orders"][order_id]}
TOOLS = {"refund": refund, "send_email": send_email,
"close_ticket": close_ticket, "get_order": get_order}
TASKS = [
{"id": "refund-and-notify",
"check": lambda w: w["orders"]["A-1"] == "refunded"
and w["emails"] == ["ada@example.com"]},
{"id": "refund-and-close",
"check": lambda w: w["orders"]["A-2"] == "refunded"
and w["tickets"]["T-2"] == "closed"},
]
# Three versions of the same agent, written out as the calls each one makes.
VERSIONS = {
"v1 baseline": {
"refund-and-notify": [("refund", {"order_id": "A-1"}),
("send_email", {"to": "ada@example.com"})],
"refund-and-close": [("refund", {"order_id": "A-2"}),
("close_ticket", {"tid": "T-2"})],
},
"v2 after prompt edit": {
"refund-and-notify": [("get_order", {"order_id": "A-1"}),
("refund", {"order_id": "A-1"}),
("get_order", {"order_id": "A-1"}),
("send_email", {"to": "ada@example.com"}),
("get_order", {"order_id": "A-1"})],
"refund-and-close": [("get_order", {"order_id": "A-2"}),
("refund", {"order_id": "A-2"}),
("get_order", {"order_id": "A-2"}),
("close_ticket", {"tid": "T-2"}),
("get_order", {"order_id": "A-2"})],
},
"v3 chattier": {
"refund-and-notify": [("refund", {"order_id": "A-1"}),
("send_email", {"to": "ada@example.com"}),
("send_email", {"to": "ada@example.com"})],
"refund-and-close": [("refund", {"order_id": "A-2"}),
("close_ticket", {"tid": "T-2"})],
},
}
IN_PER_M, OUT_PER_M = 3.00, 15.00
def cost_of(turns):
"""Same growing-context model as the budgets section."""
return sum((1200 + 700 * t) / 1e6 * IN_PER_M + 120 / 1e6 * OUT_PER_M
for t in range(turns))
def evaluate(version):
passed, turns, cost = 0, 0, 0.0
for task in TASKS:
world = copy.deepcopy(WORLD0) # every task starts from the same world
calls = VERSIONS[version][task["id"]]
for name, args in calls:
TOOLS[name](world, **args)
passed += bool(task["check"](world))
turns += len(calls) + 1 # one more model call to finish
cost += cost_of(len(calls) + 1)
return passed, turns, cost
print(f"{'version':<22}{'passed':>8}{'turns':>7}{'cost':>9}")
for version in VERSIONS:
passed, turns, cost = evaluate(version)
print(f"{version:<22}{passed:>5}/{len(TASKS)}{turns:>7} ${cost:>6.3f}")
# version passed turns cost
# v1 baseline 2/2 6 $ 0.045
# v2 after prompt edit 2/2 12 $ 0.128
# v3 chattier 1/2 7 $ 0.057Three things in that table are worth taking separately. Version two is the hidden regression: it passes exactly what the baseline passed, so a pass-rate-only eval reports no change, while it takes twice the turns and costs 0.128 against 0.045. Version three fails a task the baseline passed, and the reason it fails is the thing to notice.
The hidden regression#
Success rate flat, cost doubled
A prompt change can keep every task passing while doubling the turns taken. Nothing fails, nobody notices, and the bill quietly grows. This is exactly why cost and turn count belong in the eval output next to the pass rate.
Version three sent the customer two identical emails. It fails only because the check compares the list of emails to exactly one address. Write the same check the way most people write it and the regression disappears.
LOOSE = {"refund-and-notify": lambda w: w["orders"]["A-1"] == "refunded"
and len(w["emails"]) >= 1,
"refund-and-close": lambda w: w["orders"]["A-2"] == "refunded"
and w["tickets"]["T-2"] == "closed"}
def evaluate_with(version, checks):
passed = 0
for task in TASKS:
world = copy.deepcopy(WORLD0)
for name, args in VERSIONS[version][task["id"]]:
TOOLS[name](world, **args)
passed += bool(checks[task["id"]](world))
return passed
exact = {t["id"]: t["check"] for t in TASKS}
for version in VERSIONS:
print(f"{version:<22} exact check {evaluate_with(version, exact)}/2"
f" 'at least one email' check {evaluate_with(version, LOOSE)}/2")
# v1 baseline exact check 2/2 'at least one email' check 2/2
# v2 after prompt edit exact check 2/2 'at least one email' check 2/2
# v3 chattier exact check 1/2 'at least one email' check 2/2Agents make that mistake expensive in a way single completions do not, because doing something twice is one of their characteristic failures and a check written as at-least-once cannot see it. Assert on equality wherever you can: the list of emails, the number of rows written, the set of records touched. Where equality is genuinely too strict, assert on the count and on what was not touched, which catches an agent that refunded the right order and also the one next to it.
Two practical notes on building the harness. Run the tasks against a fresh copy of the world each time, or a task that mutates something makes every later task depend on the order they ran in, and a flaky eval is worse than no eval because it trains the team to ignore red. And run them concurrently for speed but never against shared state, which is the same rule written twice.
What to record per run#
The eval output should be a row per run, not a single number, because the aggregate hides everything you would want to act on. Keep the task id, whether it passed, turns, cost, wall-clock time, the stop reason, and the sequence of tool names. The last of those is close to free and pays for itself the first time a task starts failing: comparing the tool sequence of a failing run with the sequence from when it passed usually locates the change without reading any prose at all.
- Pass rate, with the failing task ids listed rather than counted.
- Mean and worst-case turns and cost. The mean hides the one run that took thirty turns.
- How many runs stopped on a budget rather than because the model finished.
- Tool error rate: how many tool calls returned an error, which usually moves before the pass rate does.
- Run the whole set more than once. Agents are non-deterministic, so a pass rate from a single pass over twenty tasks has a margin of several points on it.
Where people go wrong
Only evaluating the happy path. Agents fail on the strange cases, so an eval set of clean inputs measures the scenario you were never worried about.
Build this: Build a ten-task eval for an agent that reports pass rate, mean turns and mean cost in one command. Then make a change you believe is an improvement and see whether all three numbers agree with you.
Open this section on its own pageWhere you use this
A topic is easier to care about when you can see what it is for. These projects lean on it at a named stage.
- Stage 2, The loop, written by hand: What an agent is once the vocabulary is removed, and which part of it you are writing.
- Stage 5, The transcript grows: Managing the transcript as it grows, and what has to survive trimming for the run to continue.
Now actually do it
Reading a topic is not learning it. These are the hours that count, and each one tells you when you are finished.
1Write the loop from nothing
6hNo framework. A while loop with tool dispatch, message accumulation, a turn budget and a stop condition, in under 300 lines. One real tool that touches a real API.
Done when: You can draw it on paper from memory and explain why each guard exists.
2Break it deliberately
5hAdd three more tools: one that throws, one that returns a 5000 token payload, one that is slow. Make the agent survive all three, run the independent ones concurrently, and keep the context from ballooning.
Done when: A failing tool does not end the run, the context does not grow unbounded, and parallel calls actually run in parallel.
3Budgets and a stuck detector
3hAdd turn, wall-clock and cost budgets. Detect the same tool being called with identical arguments three times and intervene. Return a structured partial result when a budget runs out.
Done when: Given an impossible task it stops cleanly on each budget in turn and explains what it tried.
14 hours of practice
Words used here
2 terms this topic teaches, each with a one-line definition if one of them stopped you.
- Agent loop
- The cycle of calling a model, executing whatever tool it asked for, appending the result to the conversation, and calling again until it stops asking. Nearly every agent is this loop plus a stopping rule.
- ReAct
- Interleaving a model's written reasoning with its tool calls, so each action follows an explicit thought and each observation feeds the next one. The pattern most agent frameworks are built around.
Interview questions
12 questions, grouped by the section that teaches the answer. An answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
What an agent actually is
What is an agent, in one sentence?
A loop that calls a model, executes any tools the model requests, feeds the results back, and repeats until the model stops asking or a budget runs out.
When would you not use an agent?
When the steps are known in advance. A fixed pipeline is cheaper, faster and far easier to debug. Agents earn their cost only when the path depends on what is discovered along the way.
Then they push further
Give an example of a task that looks like it needs an agent but does not.
Summarising a support ticket and routing it to a team. It feels open-ended, but the steps are fixed: classify, summarise, route. Two calls with structured output do it cheaper, and every step can be tested on its own. An agent would add variance and cost for no gain.
And one that genuinely does?
Working out why a customer's invoice is wrong across billing, usage and plan history, where what to look up next depends on what the last lookup showed. The number and order of steps cannot be written down in advance.
What gets marked up. Strong answers name what agency costs: variance, latency, money and harder debugging, and treat an agent as something a task has to earn. Weak answers treat agents as the default modern architecture.
Where does the loop run?
In your code. The model cannot execute anything or call itself. It only emits a structured request, and your program decides whether to act on it and what to send back.
Budgets, loops and giving up well
How do you stop an agent running forever?
Three independent budgets: turns, wall-clock time and cost. Any one being exceeded stops the run. Turns alone are not enough because a single turn can be slow or expensive.
Then they push further
Why is a turn limit alone not enough?
One turn can call a slow tool or produce a huge output, so a run can blow its time or cost budget in very few turns. Each budget protects against a different failure.
What does the user see when a budget trips?
A structured partial result: what was done, what was not, and why it stopped, with a way to continue if that makes sense. A silent failure or a generic error throws away the work already done.
How would you choose the limits?
From data. Run the eval tasks, look at turns, time and cost for the runs that succeeded, and set limits a little above the high end of that range. Revisit them when tools or models change.
What gets marked up. Marked up for independent budgets, for returning partial results instead of throwing, and for deriving limits from measured runs rather than picking round numbers.
An agent keeps calling the same tool with the same arguments. What do you do?
Detect the repeat by hashing the tool name and arguments, and when it repeats, return a result telling the model it is stuck and should try something else. Killing the run is the fallback if that does not work.
What should an agent return when it runs out of budget?
A structured partial result: what it achieved, what it could not, why it stopped, and what it cost. Raising an exception throws away work the caller could still use.
Managing the context as it grows
An agent overflows its context after ten turns. What do you do?
In order: make tools return less, drop stale tool results, summarise older turns while keeping recent ones verbatim, and offload large payloads to storage with an id the model can use to retrieve them. Moving to a bigger context window is the last resort.
Then they push further
Which of those usually gives the biggest win?
Making tools return less. Tool results are usually most of the context, and a tool that returns a whole record when the model needs three fields fills the window fast. Trimming at the source also cuts cost on every later turn.
What is the risk of summarising older turns?
The summary drops something that matters later: an id, a constraint the user stated, or a failed attempt the agent will now repeat. Keep a structured record of decisions and key facts outside the summary, and keep recent turns word for word.
What gets marked up. Strong answers attack the source of the bloat before compressing, and name what compaction loses. A bigger context window as the first answer is marked down.
What must survive compaction?
The original task, anything the user said including corrections and constraints, decisions already taken, and failures already hit. Losing those causes the agent to wander or repeat work.
How does prompt caching interact with context management?
Caching matches a stable prefix, so anything volatile must go last. If a summary is inserted near the front, it invalidates the cache every time it updates. Order from most stable to most volatile and the cache keeps working.
Knowing whether it works
How do you evaluate an agent rather than a single completion?
Check the end state of the world rather than the text, because the agent's job is to do something. Then track efficiency alongside correctness: turns and cost per task. A run that succeeds after fourteen turns is not equivalent to one that succeeds in three.
Your agent's success rate is unchanged after a prompt edit. Is it safe to ship?
Not necessarily. Check turns and cost as well. A change can hold the pass rate while doubling spend, and nothing in a pass-rate-only eval will tell you.
How many eval tasks do you need?
Twenty real ones beats two hundred synthetic ones. Draw them from actual usage, include every failure you have already seen in production, and version the set so a score from last month still means something.
Resources
Free to watch and read, picked for this topic.
Videos
For what an agent actually is
For budgets, loops and giving up well
For managing the context as it grows
For knowing whether it works
Full courses: the whole topic, start to finish
Read and explore
Stop 16 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.