Context management

A model has no memory between calls. Everything it knows about your conversation — the system prompt, every earlier turn, the tool definitions, every tool result — is resent on each request, and all of it has to fit in one bounded space called the context window, measured in tokens. Context management is the work of deciding what goes in that space when the material outgrows it, which for a long conversation or a working agent it always does. This section is about staying inside the window without losing the thread: measuring what is actually filling the window, then the three ways to make space — dropping old turns, replacing them with a summary, or moving detail somewhere the model can fetch back — each written out as code you can lift. The judgement is in what must never be dropped, whatever else goes.

Measure before you cut#

Almost everyone trims the wrong thing, because the intuition about what is large is wrong. Tool results are usually the biggest item by a wide margin and the least valuable to keep once they have been acted on. Count before deciding.

def audit(messages, system, tools, count_tokens):
    """Where the window is actually going, largest first."""
    rows = [
        ("system", count_tokens(system)),
        ("tool definitions", count_tokens(str(tools))),
    ]
    by_role = {}
    for m in messages:
        by_role[m["role"]] = by_role.get(m["role"], 0) + count_tokens(str(m["content"]))
    rows += sorted(by_role.items(), key=lambda kv: -kv[1])

    total = sum(n for _, n in rows)
    for label, n in sorted(rows, key=lambda kv: -kv[1]):
        print(f"{label:20} {n:7,}  {n / total:5.1%}")
    return total

# system                 420   1.2%
# tool                22,100  63.4%   <- the whole problem
# user                 6,300  18.1%
# assistant            5,030  14.4%

Trimming: drop the oldest, keep the shape#

The cheapest technique, and the one with a trap. You cannot simply slice the list: the API rejects a tool result whose matching tool call has been trimmed away, and a conversation that starts on an assistant turn is malformed. Trim in whole exchanges, and keep the system prompt pinned.

def trim(messages, count_tokens, budget, keep_recent=6):
    """Keep the most recent exchanges that fit, without orphaning tool results."""
    kept, used = [], 0
    for m in reversed(messages):
        size = count_tokens(str(m["content"]))
        if len(kept) >= keep_recent and used + size > budget:
            break
        kept.append(m)
        used += size
    kept.reverse()

    # A tool result whose tool_use is gone will be rejected by the API.
    call_ids = {
        c["id"]
        for m in kept if isinstance(m["content"], list)
        for c in m["content"] if c.get("type") == "tool_use"
    }
    kept = [
        m for m in kept
        if not (isinstance(m["content"], list)
                and any(c.get("type") == "tool_result"
                        and c.get("tool_use_id") not in call_ids
                        for c in m["content"]))
    ]

    # A conversation may not begin on an assistant turn.
    while kept and kept[0]["role"] != "user":
        kept.pop(0)
    return kept

Summarising: fold the old turns into one#

When the early conversation still matters, replace it rather than dropping it. Summarise everything older than the recent window into a single message and carry that forward. The summary must be about decisions and state, not narrative, or it will read pleasantly and be useless to the model.

SUMMARISE = """Summarise the conversation below for an assistant that will
continue it and cannot see the original. Record only:
- decisions made and constraints agreed, verbatim where exact wording matters
- facts established, with the identifiers, names and numbers involved
- what has been tried and failed, so it is not tried again
- what the user is currently trying to do

Omit pleasantries, restated questions and anything already superseded.
Write it as notes, not prose."""

def compact(client, messages, count_tokens, budget, keep_recent=6):
    if count_tokens(str(messages)) <= budget:
        return messages

    old, recent = messages[:-keep_recent], messages[-keep_recent:]
    if not old:
        return messages

    summary = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        system=SUMMARISE,
        messages=[{"role": "user", "content": str(old)}],
    ).content[0].text

    return [
        {"role": "user", "content": f"Summary of the conversation so far:
{summary}"},
        {"role": "assistant", "content": "Understood, I have that context."},
        *recent,
    ]

Offloading: keep a reference, not the content#

The best option for tool results, which dominate the audit above. Write the full result somewhere the model can read back on request, and leave a short handle in the conversation. The model keeps the ability to consult it without paying for it on every turn.

STORE = {}

def offload(result, keep=200):
    """Replace a large tool result with a stub the model can expand."""
    text = str(result)
    if len(text) <= keep * 2:
        return text
    ref = f"res_{len(STORE)}"
    STORE[ref] = text
    return (
        f"{text[:keep]}
"
        f"[truncated, {len(text):,} chars total. "
        f"Call read_result(ref={ref!r}, start=..., length=...) for the rest.]"
    )

def read_result(ref, start=0, length=2000):
    """Expose this as a tool so the model can pull back what it needs."""
    return STORE[ref][start : start + length]

Order it so the cache still hits#

Every technique here rewrites history, and prefix caching matches from the very first character, so a careless compaction invalidates the cache on every turn and costs more than it saves. Put the things that never change first — system prompt, tool definitions, the stable summary — and confine edits to the tail.

system prompt          never changes        cached
tool definitions       never changes        cached
summary of old turns   changes rarely       cached until it does
recent exchanges       changes every turn   not cached
current question       changes every turn   not cached

What must never be dropped#

• The system prompt. Trimming it produces an agent that forgets its own constraints and no error to explain why.
• Anything the user explicitly asked to be remembered. Losing it is the failure people never forgive.
• Decisions and their reasons. An agent that re-litigates a settled choice looks broken.
• The most recent exchange, whatever the budget says. Without it there is no conversation.
• Identifiers: file paths, ticket numbers, order references. They are short and irreplaceable.

Compaction keeps one conversation affordable. It does nothing for the next one, which starts again knowing nothing about the user. This section is about what an assistant should carry between sessions and how: storing settled facts rather than transcripts, retrieving only what is relevant to the request in hand, and letting people see and correct what has been remembered about them. Memory nobody can inspect produces an assistant that is confidently wrong for reasons nobody can trace.

Memory is three parts, and none of them is a model feature#

A model has no memory at all. What a product calls memory is a system you build around it, and it has exactly three parts: somewhere durable to keep things, a write policy deciding what gets stored and what it replaces, and a retrieval policy deciding which of those things enter the next prompt. The model contributes nothing to any of it. It sees a block of text in its context and treats it as true, whether the retrieval policy chose well or badly.

storage            rows somewhere durable: a table, a file, a vector index
write policy       what becomes a row, when, and what it supersedes
retrieval policy   which rows enter the next prompt, under what budget

the model          sees only what the retrieval policy selected, and
                   cannot tell a correct memory from a wrong one

The asymmetry between the two policies is what makes the design hard. Writing is cheap, unbounded and permanent: nothing stops you appending a row on every turn forever. Retrieval is bounded and paid for on every single request, because whatever comes back is tokens in the window alongside the system prompt, the tools and the conversation itself. A store that grows without limit is only a storage bill; a retrieval policy that returns the wrong three rows is a wrong answer delivered confidently, and that is the failure users notice.

Two kinds#

• Short-term: the current conversation, managed with the trimming and summarising techniques in the previous section.
• Long-term: facts that should survive across sessions, such as preferences and past decisions, stored outside the context and retrieved when relevant.

The two are not variations on one mechanism, and the boundary between them is a write policy in its own right. Compaction, covered in the window section of this topic, decides what survives within a session and is allowed to be lossy, because the original transcript is still on disk and the conversation is still in front of the user. Long-term memory decides what survives the session ending, is not lossy in the same forgiving way, and will be read back weeks later by an assistant that has no transcript to check it against. Promoting something from one to the other is a decision, and the moment to make it is when the user states something rather than when the session ends.

What is worth remembering#

Storing everything recreates the original problem at a larger scale. Store stable facts and decisions, not the conversation itself: the user prefers metric units, the project uses PostgreSQL, the tone should be formal.

• Preferences: units, tone, language, formatting. Few, small and almost always relevant, which makes them cheap to send on every request rather than retrieve.
• Facts: settled properties of the user's world. The database is PostgreSQL 16, the team is four people, the deploy target is Frankfurt.
• Decisions: choices made and the reason, so the assistant does not reopen a settled argument. These are the ones users are most annoyed to lose.
• Identifiers: repository names, ticket prefixes, account numbers. Short, irreplaceable, and useless if the assistant guesses instead.
• Episodic transcripts: what was said, verbatim, with timestamps. Store these if you must for audit, and keep them out of the retrieval path by default, because they are large and they contain every superseded version of everything.

The category to be suspicious of is anything transient. A user who says they are in a hurry is telling you about this minute, not about themselves, and an assistant that is still rushing three weeks later is a bug with no error message. The same goes for anything the model inferred rather than the user stated: an inference stored as a fact is indistinguishable from a fact once it is a row in a table, and it will be read back with the same confidence.

The store, and the write policy that bounds it#

The single decision that makes a memory store work is keying a row by what it is about rather than by when it was written. A store keyed by turn is an append-only log, so it grows with conversation length and holds every superseded version of every fact. A store keyed by subject has one slot per subject, so a new value overwrites the old one, growth stops at the number of distinct things the user has told you, and staleness cannot accumulate because there is nowhere for it to sit.

import re
from dataclasses import dataclass

@dataclass
class Memory:
    key: str        # what this memory is ABOUT: kind plus subject
    text: str       # the settled fact, written to stand on its own
    turn: int       # when it was last written or reconfirmed
    source: str     # the user message it came from, so it can be shown back
    seen: int = 1   # how many times the user has restated it

class MemoryStore:
    """Storage plus a write policy. One slot per subject, not one row per turn."""

    DURABLE = {"preference", "fact", "decision", "identifier"}

    def __init__(self):
        self.slots, self.log = {}, []

    def write(self, kind, subject, text, turn, source):
        if kind not in self.DURABLE:
            return self._note(turn, "rejected", f"{kind} is not durable")
        key = f"{kind}|{subject}"
        old = self.slots.get(key)
        self.slots[key] = Memory(key, text, turn, source,
                                 seen=old.seen + 1 if old and old.text == text else 1)
        if old is None:
            return self._note(turn, "new", key)
        if old.text == text:
            return self._note(turn, "reconfirmed", key)
        return self._note(turn, "superseded", f"{key}: {old.text} -> {text}")

    def _note(self, turn, action, detail):
        self.log.append((turn, action, detail))
        return action

store = MemoryStore()
transcript = [
    (1, "preference", "units", "Prefers metric units.", "give me that in metric please"),
    (2, "fact", "atlas.db", "Atlas uses a PostgreSQL 16 database.", "atlas is on postgres 16"),
    (3, "mood", "now", "Is in a hurry.", "quick, I am late for a meeting"),
    (4, "preference", "units", "Prefers metric units.", "metric again, thanks"),
    (5, "decision", "atlas.auth", "Atlas authenticates with session cookies, not JWTs.",
     "we settled on session cookies"),
    (6, "fact", "beacon.db", "Beacon uses a MySQL 8 database.", "beacon is mysql 8"),
    (7, "preference", "units", "Prefers imperial units, feet and inches.",
     "actually switch me to feet and inches"),
]
for turn, kind, subject, text, source in transcript:
    store.write(kind, subject, text, turn, source)

for turn, action, detail in store.log:
    print(f"turn {turn}  {action:12s} {detail}")
print(f"\n{len(transcript)} candidates -> {len(store.slots)} stored memories")

# turn 1  new          preference|units
# turn 2  new          fact|atlas.db
# turn 3  rejected     mood is not durable
# turn 4  reconfirmed  preference|units
# turn 5  new          decision|atlas.auth
# turn 6  new          fact|beacon.db
# turn 7  superseded   preference|units: Prefers metric units. -> Prefers imperial units, feet and inches.
#
# 7 candidates -> 4 stored memories

Four things in that class are worth naming out loud. The kind allow-list is the write policy's filter, and rejecting the mood is what stops transient state becoming permanent. The composite key is what makes superseding automatic rather than a separate reconciliation job you will never write. The seen counter distinguishes something the user has said once from something they have insisted on three times, which is the signal to use when two memories conflict and you have to pick. And the source field holds the original message, which is what you show a user who asks why the assistant thinks that, and what you show yourself when a memory turns out to be wrong.

The part this class does not do is decide what counts as a candidate in the first place. In a real system a small model reads each turn and proposes zero or more candidate writes as structured output, and the class above is what then accepts or rejects them. Keeping those two steps separate matters: extraction is fuzzy and worth a cheap model, and the policy is a handful of deterministic rules you can unit-test. A system that lets the extractor write straight to storage has no write policy, only an extractor.

The retrieval policy#

Retrieval is where the budget bites. The memories that come back are tokens in the same window as everything else, so the policy needs three things in order: a scope filter, so memories belonging to another project or another account cannot be returned at all; a relevance score with a floor, so a request that matches nothing gets nothing; and a hard budget, so the worst case is bounded rather than typical.

STOP = {"a", "an", "and", "as", "can", "do", "does", "for", "how", "i", "in", "is",
        "it", "its", "me", "my", "not", "of", "on", "or", "our", "should", "that",
        "the", "this", "to", "we", "what", "which", "with", "you"}

def terms(text):
    out = set()
    for w in re.findall(r"[a-z0-9]+", text.lower()):
        w = w[:-1] if w.endswith("s") and len(w) > 3 else w   # crude stemming
        if w not in STOP:
            out.add(w)
    return out

def retrieve(store, request, scope, budget_chars=280, floor=0.30):
    """Retrieval policy: scope, then always-on, then relevance, under one budget."""
    q = terms(request)
    always, scored = [], []
    for m in store.slots.values():
        kind, subject = m.key.split("|", 1)
        owner = subject.split(".")[0] if "." in subject else None
        if owner is not None and owner != scope:
            continue                                   # belongs to another project
        if kind == "preference":
            always.append(m)                           # few, small, always relevant
            continue
        t = terms(m.text)
        scored.append((len(q & t) / len(q) if q else 0.0, m))

    scored.sort(key=lambda row: -row[0])
    kept, used = list(always), sum(len(m.text) for m in always)
    for score, m in scored:
        if score < floor or used + len(m.text) > budget_chars:
            break
        kept.append(m)
        used += len(m.text)
    return kept, scored

for request, scope in [("what database does atlas use?", "atlas"),
                       ("how tall is the Eiffel tower?", "atlas"),
                       ("what database does beacon use?", "beacon")]:
    kept, scored = retrieve(store, request, scope)
    print(f"\n{request!r}   scope={scope}")
    for score, m in scored:
        print(f"   {'IN ' if m in kept else 'out'} relevance {score:.2f}  {m.text}")
    print(f"   sent: {len(kept)} memories, {sum(len(m.text) for m in kept)} chars")

#
# 'what database does atlas use?'   scope=atlas
#    IN  relevance 0.75  Atlas uses a PostgreSQL 16 database.
#    out relevance 0.25  Atlas authenticates with session cookies, not JWTs.
#    sent: 2 memories, 76 chars
#
# 'how tall is the Eiffel tower?'   scope=atlas
#    out relevance 0.00  Atlas uses a PostgreSQL 16 database.
#    out relevance 0.00  Atlas authenticates with session cookies, not JWTs.
#    sent: 1 memories, 40 chars
#
# 'what database does beacon use?'   scope=beacon
#    IN  relevance 0.75  Beacon uses a MySQL 8 database.
#    sent: 2 memories, 71 chars

The middle case is the one to look at. A question about the Eiffel tower returns one memory, the always-on preference, and nothing else, because the floor rejected everything below 0.30. That empty result is correct behaviour and it is the behaviour most implementations get wrong, because a top-k search always returns k things however irrelevant they are. Top-k with no floor is how an assistant ends up being told about your database while you are asking about a tower, and the model has no way to know the context it was given is noise.

Splitting preferences out from retrieved facts is the other decision worth defending. Preferences are a handful of rows, they are short, and they are relevant to nearly every request, so paying to retrieve them is more expensive than paying to send them. Facts are the opposite: many, longer, and relevant to a few requests each. A single ranked list treats both the same and so either floods the window with preferences or drops them on the requests where they mattered. The budget itself belongs in the same accounting as everything else in the window; the window section in this topic has the audit that shows where the tokens are actually going.

The three ways memory goes wrong#

Stale facts that contradict newer ones, unbounded growth, and confidently retrieving the wrong memory. They sound like three problems and they are close to one: all three follow from a store that records when something was said instead of what it was about.

import random

rng = random.Random(0)
SUBJECTS = [f"subject{i:02d}" for i in range(40)]

append_only, slotted = [], {}
for turn in range(1, 2001):
    s = rng.choice(SUBJECTS)
    text = f"{s} is set to value{rng.randrange(5)}."
    append_only.append((turn, s, text))
    slotted[s] = (turn, s, text)

print(f"2000 turns over {len(SUBJECTS)} subjects")
print(f"  append-only rows {len(append_only):5d}   {sum(len(t) for _, _, t in append_only):6d} chars")
print(f"  one slot each    {len(slotted):5d}   {sum(len(t) for _, _, t in slotted.values()):6d} chars")

versions = [t for _, s, t in append_only if s == SUBJECTS[0]]
print(f"\neverything the store knows about {SUBJECTS[0]}")
print(f"  append-only: {len(versions)} memories, {len(set(versions))} mutually exclusive")
print(f"  one slot:    1 memory, {slotted[SUBJECTS[0]][2]}")

# The third failure: a fact whose subject was never recorded.
flat = MemoryStore()
flat.write("fact", "db", "Uses a PostgreSQL 16 database.", 1, "atlas is on postgres 16")
kept, scored = retrieve(flat, "what database does beacon use?", scope="beacon")
print("\nasked about Beacon, given a memory that never said which project it was about")
for score, m in scored:
    print(f"  relevance {score:.2f}  {m.text}   -> {'sent' if m in kept else 'held back'}")

# 2000 turns over 40 subjects
#   append-only rows  2000    54000 chars
#   one slot each       40     1080 chars
#
# everything the store knows about subject00
#   append-only: 51 memories, 5 mutually exclusive
#   one slot:    1 memory, subject00 is set to value3.
#
# asked about Beacon, given a memory that never said which project it was about
#   relevance 0.50  Uses a PostgreSQL 16 database.   -> sent

Staleness first. The append-only store holds fifty-one memories about subject00 and five of them cannot all be true. Retrieval scores by relevance, so all five score identically, and whichever ones fit the budget go into the prompt. The model is then given a contradiction with no way to resolve it, and what it does with that is not predictable. If you have to keep history, give each row a validity flag and retrieve only the current one, so the audit trail lives in storage rather than in the retrieval path.

Growth second. Fifty times the rows for the same knowledge, and the ratio keeps rising because the subject count is fixed by the user's world while the turn count is not. Unbounded growth rarely shows up as a storage bill. It shows up as retrieval getting slower and less precise, because every query now ranks against thousands of near-duplicate rows and the top results are all versions of the same thing.

Confident wrong retrieval third, and it is the one that damages trust. The memory in the last block is a true sentence: something does use PostgreSQL 16. It was written without recording which project it was about, so it matches a question about a different project at 0.50 and is sent. Nothing in the pipeline errors, the model has no reason to doubt it, and the user gets a specific, wrong, fluent answer. The fix is in the write policy rather than the retrieval one: a fact whose subject you cannot name is a fact you should not store.

Show it, and let people delete it#

Everything above assumes you can inspect the store, which is why the source field exists on the record. A memory view is a list of rows with what is remembered, when it was written, the message it came from, and a delete button, and it is perhaps an afternoon of work. Without it a wrong memory is undiagnosable from the outside and unfixable from the inside: the user sees an assistant that is oddly wrong about them and has no way to say why, and you see a support ticket that reproduces for one account and no other.