Managing the context as it grows
By the end: Your agent can run for many turns without overflowing or becoming unaffordable.
- Why context grows faster than you expect
- Trimming, summarising and offloading
- Keeping the system prompt stable for caching
- What must never be dropped
New to the agent loop? Agent is a loaded word for something small. Call the model; if the reply asks to use a tool, run the tool, add the result to the conversation, and call the model again. Keep going until it stops asking or until you decide it has had enough turns. That loop, a few dozen lines of ordinary code, is the agent — there is no other machinery hiding behind the word. Why it matters
Every turn an agent takes adds to the pile of text resent on the next one, so a run that starts cheap becomes expensive and eventually will not fit at all. This section is about deciding what stays and what goes: trimming tool results, summarising old turns, moving bulk out to storage, and ordering the prompt so none of that destroys your caching. It also covers what must survive any amount of compaction, because an agent that has lost sight of what it was asked does not stop, it carries on doing something else.
The growth problem#
Every turn adds the model's message and every tool result, and all of it is resent next turn. Tool results are usually the bulk of it. An agent that reads three documents and searches twice can be carrying tens of thousands of tokens by turn six.
Turn 1: 2,400 tokens
Turn 3: 9,100 tokens
Turn 6: 31,000 tokens same task, nothing unusual happenedIt is worth seeing where that comes from rather than taking it on trust, because the answer decides which fix is worth building. The run below is twelve turns alternating a search and a document read, with the tools returning what tools return by default: the whole search response and the whole page. Token counts use characters over four, which is not your provider's tokeniser but is deterministic and close enough to reason about.
import json
def tokens(s):
"""Characters over four. Not your provider's tokeniser, but deterministic,
close enough to reason with, and it makes this block reproducible."""
return max(1, len(s) // 4)
PARA = ("Refunds on the annual plan are available within fourteen days of the "
"renewal date, and are returned to the original payment method within "
"five working days of approval. Partial refunds are calculated pro rata "
"from the cancellation date. ")
DOCUMENT = {"doc_id": "POL-14", "title": "Refunds and cancellations",
"text": PARA * 40} # a policy page, in full
SEARCH_HITS = [{"order_id": "A-44" + str(70 + i), "customer_id": "C-118",
"status": "shipped", "total_cents": 2900 + i,
"snippet": PARA[:220] + " (order A-44" + str(70 + i) + ")"}
for i in range(20)]
SCRIPT = [("search_orders", SEARCH_HITS), ("read_document", DOCUMENT)] * 6
SYSTEM = 900 # tokens of system prompt and tool definitions
ASSISTANT = 120 # the model's own message each turn
IN_PER_M = 3.00
def slim(name, result):
"""What the tool should have returned: the fields the model can act on."""
if name == "search_orders":
return [{"order_id": r["order_id"], "status": r["status"]} for r in result[:5]]
return {"doc_id": result["doc_id"], "title": result["title"],
"text": result["text"][:400] + "..."}
def run(policy, keep_recent=3):
context, peak, billed, store = [], 0, 0, {}
for turn, (name, result) in enumerate(SCRIPT):
if policy == "trim in the tool":
body = json.dumps(slim(name, result))
elif policy == "offload":
key = name + "-" + str(turn)
store[key] = result
body = json.dumps({"stored_as": key,
"rows": len(result) if isinstance(result, list) else 1})
else:
body = json.dumps(result)
context.append({"turn": turn, "name": name, "body": body})
if policy == "drop stale results":
for e in context[:-keep_recent]:
e["body"] = json.dumps({"note": e["name"] + " ran at turn "
+ str(e["turn"]) + ", result dropped"})
size = SYSTEM + sum(tokens(e["body"]) + ASSISTANT for e in context)
peak = max(peak, size)
billed += size
return peak, billed, billed / 1e6 * IN_PER_M
print("one search_orders result:", tokens(json.dumps(SEARCH_HITS)), "tokens ->",
tokens(json.dumps(slim("search_orders", SEARCH_HITS))), "trimmed")
print("one read_document result:", tokens(json.dumps(DOCUMENT)), "tokens ->",
tokens(json.dumps(slim("read_document", DOCUMENT))), "trimmed")
print()
print(f"{'policy':<22}{'peak context':>13}{'billed in':>11}{'cost':>9}")
for policy in ("as written", "drop stale results", "trim in the tool", "offload"):
peak, billed, cost = run(policy)
print(f"{policy:<22}{peak:>13,}{billed:>11,} ${cost:>6.3f}")
# one search_orders result: 1700 tokens -> 56 trimmed
# one read_document result: 2387 tokens -> 118 trimmed
#
# policy peak context billed in cost
# as written 26,862 177,492 $ 0.532
# drop stale results 8,931 87,837 $ 0.264
# trim in the tool 3,384 26,760 $ 0.080
# offload 2,467 20,983 $ 0.063The first two lines are the whole argument. A search returning twenty hits with a snippet each is 1,700 tokens, and the five order ids the model will actually act on are 56. You are paying thirty times over for fields nobody reads, and paying it again on every subsequent turn. That is why the ordered list below starts where it does.
Four things to do about it#
- 1.Return less in the first place. Summarise or truncate inside the tool, before the result ever enters the context. Cheapest fix by far.
- 2.Drop old tool results. A search result from turn two is rarely needed at turn nine. Replace it with a one-line note that it happened.
- 3.Summarise older turns into a running summary, keeping recent turns verbatim.
- 4.Offload to storage. Write the full result to a file or database and keep only an identifier the model can use to fetch it again.
def compact(messages, keep_recent=6):
if len(messages) <= keep_recent + 1:
return messages
system, old, recent = messages[0], messages[1:-keep_recent], messages[-keep_recent:]
summary = client.chat([
{"role": "user",
"content": "Summarise this agent transcript. Keep every decision, "
"finding and open question. Drop the raw tool output.\n\n"
+ render(old)},
]).text
return [system, {"role": "user", "content": f"Earlier:\n{summary}"}, *recent]Keep the front of the prompt stable#
Caching is prefix-matched
Providers cache the processed form of a stable prompt prefix, which makes repeated calls much cheaper. It matches from the start, so one changed byte early invalidates everything after it. Put a timestamp in your system prompt and you have silently switched off your largest cost saving.
Order everything from most stable to most volatile:
system prompt never changes
tool definitions rarely change
few-shot examples rarely change
summarised history changes slowly
recent turns changes every turn
current question changes every turnAgent loops are the best case for a cache, because the message list only ever grows at the end. Everything sent on turn five was sent on turn four in exactly the same order, so on a well-behaved loop the entire previous context is a cache hit and only the newest turn is fresh. Pricing that out on the same twelve-turn run says how much the discipline is worth, and what one careless line costs.
# Assumed prices: a cache read costs a tenth of a fresh input token, and
# writing to the cache costs a quarter more. Check your provider's numbers.
READ, WRITE = 0.10, 1.25
def sizes(policy, keep_recent=3):
"""Context size at each turn, using the same machinery as above."""
out, context = [], []
for turn, (name, result) in enumerate(SCRIPT):
body = json.dumps(slim(name, result) if policy == "trim in the tool" else result)
context.append({"turn": turn, "name": name, "body": body})
out.append(SYSTEM + sum(tokens(e["body"]) + ASSISTANT for e in context))
return out
def cost(policy, stable_prefix=True, compact_every=None):
"""Append-only messages mean everything before this turn is a cache hit."""
sz, total, cached = sizes(policy), 0.0, 0
for turn, size in enumerate(sz):
if not stable_prefix: # a timestamp in the system prompt
cached = 0
if compact_every and turn and turn % compact_every == 0:
cached = 0 # the middle was rewritten
fresh = size - cached
total += (cached * READ + fresh * WRITE) / 1e6 * IN_PER_M
cached = size
return total
for policy in ("as written", "trim in the tool"):
plain = run(policy)[2]
print(f"{policy:<18} no cache ${plain:.3f} cached ${cost(policy):.3f} "
f"cached but with a timestamp in the system prompt "
f"${cost(policy, stable_prefix=False):.3f}")
print("compacting every 4 turns, as written: $%.3f" % cost("as written", compact_every=4))
# as written no cache $0.532 cached $0.146 cached but with a timestamp in the system prompt $0.666
# trim in the tool no cache $0.080 cached $0.020 cached but with a timestamp in the system prompt $0.100
# compacting every 4 turns, as written: $0.242That third column is the line to remember. A cache you keep invalidating is not neutral, it is a 25% surcharge, because you pay the write premium on every turn and never collect a read. The same is true of anything else that moves at the front: a session id, a list of tools ordered by a dictionary that does not preserve order, a few-shot example chosen at random per call.
The last line is the tension nobody mentions. Compaction rewrites the middle of the message list, so the turn after a compaction is a complete cache miss. Compacting every four turns on this run costs 0.242 against 0.146 for leaving it alone, which means on a twelve-turn run compaction lost money. It starts winning when the context is large enough that the saved tokens outweigh the lost cache hits, or when you were going to overflow the window regardless. Compact on a size threshold, not on a turn count, and measure whether it paid.
What not to drop#
- The original task. Agents that lose sight of what they were asked wander.
- Anything the user explicitly said, especially constraints and corrections.
- Decisions already made, so the agent does not reverse them.
- Failures already encountered, or it will retry them.
That list is easy to agree with and hard to implement, because a summariser does not know which of the things it is reading will matter in four turns. The honest way to see the problem is to write down what a later turn will need, compact, and then check. Below, the summariser is a stand-in that keeps the gist and the last three turns; real ones drop different things, and they all drop something.
TURNS = [
"User: refund my last order, but to the original card only, not store credit.",
"search_orders(customer=C-118) -> 3 orders, most recent A-4471",
"get_order(A-4471) -> total 2900 cents, status shipped",
"refund(A-4471, method=store_credit) -> ERROR: customer opted out of credit",
"User: I said the card. Also the address on file is wrong, use 14 Bridge Street.",
"get_customer(C-118) -> card ending 4242, address 9 Mill Road",
"update_address(C-118, '14 Bridge Street') -> ok",
"search_policy('partial refund') -> pro rata from the cancellation date",
]
# The facts a later turn will need. Written out so the check is honest about
# what it is testing rather than grading a summary on vibes.
MUST_SURVIVE = {
"original card, not store credit": "original card",
"the order is A-4471": "A-4471",
"store credit already failed": "opted out",
"address corrected to Bridge St": "Bridge Street",
}
def naive_summary(turns, keep=3):
"""A stand-in for a summarising model that keeps the gist and the recent
turns. Real summarisers drop different things; they all drop something."""
return "Earlier: the user asked about a refund. " + " ".join(turns[-keep:])
def ledger(turns):
"""Facts are extracted as they appear and carried verbatim, not summarised."""
facts = [t for t in turns if any(m in t for m in MUST_SURVIVE.values())]
return "Known facts:\n" + "\n".join("- " + f for f in facts)
for name, text in (("naive summary", naive_summary(TURNS)), ("fact ledger", ledger(TURNS))):
kept = [label for label, needle in MUST_SURVIVE.items() if needle in text]
lost = [label for label in MUST_SURVIVE if label not in kept]
print(f"{name:<15}{len(kept)}/{len(MUST_SURVIVE)} survived lost: {lost}")
print(f"{'':<15}{len(text) // 4} tokens")
print("the eight turns in full:", len("\n".join(TURNS)) // 4, "tokens")
# naive summary 1/4 survived lost: ['original card, not store credit', 'the order is A-4471', 'store credit already failed']
# 54 tokens
# fact ledger 4/4 survived lost: []
# 105 tokens
# the eight turns in full: 131 tokensRead the lost list rather than the score. It dropped the constraint the user repeated after the agent got it wrong, and it dropped the record of that failure, which between them guarantee the agent tries store credit again. That is the specific way compacted agents go into a loop: compaction removes the memory of the thing that failed, so the same thing is tried, fails, and is compacted away once more.
So do not rely on a summary to carry the facts. Keep a small structured ledger alongside the messages, appended to as turns happen: the task as stated, every constraint the user gave, every decision taken, every tool call that failed and why. It is a few hundred tokens, it never gets summarised, and it goes in the prompt after the stable prefix. The summary is then free to be lossy, because nothing depends on it.
Where people go wrong
Summarising with the same expensive model you are running the agent with. Compaction can quietly cost more than the work. Use a cheaper model for summarising, and measure it.
If this comes up in an interview
An agent overflows its context after ten turns. What do you do?
In order: make tools return less, drop stale tool results, summarise older turns while keeping recent ones verbatim, and offload large payloads to storage with an id the model can use to retrieve them. Moving to a bigger context window is the last resort.
Then they push further
Which of those usually gives the biggest win?
Making tools return less. Tool results are usually most of the context, and a tool that returns a whole record when the model needs three fields fills the window fast. Trimming at the source also cuts cost on every later turn.
What is the risk of summarising older turns?
The summary drops something that matters later: an id, a constraint the user stated, or a failed attempt the agent will now repeat. Keep a structured record of decisions and key facts outside the summary, and keep recent turns word for word.
What gets marked up. Strong answers attack the source of the bloat before compressing, and name what compaction loses. A bigger context window as the first answer is marked down.
What must survive compaction?
The original task, anything the user said including corrections and constraints, decisions already taken, and failures already hit. Losing those causes the agent to wander or repeat work.
How does prompt caching interact with context management?
Caching matches a stable prefix, so anything volatile must go last. If a summary is inserted near the front, it invalidates the cache every time it updates. Order from most stable to most volatile and the cache keeps working.
Build this: Run an agent until it overflows, then add compaction and measure three things before and after: total tokens, total cost, and whether the task still succeeds. The third is the one people forget.
Watch it explained
This is one section of The agent loop, 14 hours across 4 sections. Read it in full to get the whole topic in order.
Stop 16 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.