Memory across sessions

Compaction keeps one conversation affordable. It does nothing for the next one, which starts again knowing nothing about the user. This section is about what an assistant should carry between sessions and how: storing settled facts rather than transcripts, retrieving only what is relevant to the request in hand, and letting people see and correct what has been remembered about them. Memory nobody can inspect produces an assistant that is confidently wrong for reasons nobody can trace.

Memory is three parts, and none of them is a model feature#

A model has no memory at all. What a product calls memory is a system you build around it, and it has exactly three parts: somewhere durable to keep things, a write policy deciding what gets stored and what it replaces, and a retrieval policy deciding which of those things enter the next prompt. The model contributes nothing to any of it. It sees a block of text in its context and treats it as true, whether the retrieval policy chose well or badly.

storage            rows somewhere durable: a table, a file, a vector index
write policy       what becomes a row, when, and what it supersedes
retrieval policy   which rows enter the next prompt, under what budget

the model          sees only what the retrieval policy selected, and
                   cannot tell a correct memory from a wrong one

The asymmetry between the two policies is what makes the design hard. Writing is cheap, unbounded and permanent: nothing stops you appending a row on every turn forever. Retrieval is bounded and paid for on every single request, because whatever comes back is tokens in the window alongside the system prompt, the tools and the conversation itself. A store that grows without limit is only a storage bill; a retrieval policy that returns the wrong three rows is a wrong answer delivered confidently, and that is the failure users notice.

Two kinds#

• Short-term: the current conversation, managed with the trimming and summarising techniques in the previous section.
• Long-term: facts that should survive across sessions, such as preferences and past decisions, stored outside the context and retrieved when relevant.

The two are not variations on one mechanism, and the boundary between them is a write policy in its own right. Compaction, covered in the window section of this topic, decides what survives within a session and is allowed to be lossy, because the original transcript is still on disk and the conversation is still in front of the user. Long-term memory decides what survives the session ending, is not lossy in the same forgiving way, and will be read back weeks later by an assistant that has no transcript to check it against. Promoting something from one to the other is a decision, and the moment to make it is when the user states something rather than when the session ends.

What is worth remembering#

Storing everything recreates the original problem at a larger scale. Store stable facts and decisions, not the conversation itself: the user prefers metric units, the project uses PostgreSQL, the tone should be formal.

• Preferences: units, tone, language, formatting. Few, small and almost always relevant, which makes them cheap to send on every request rather than retrieve.
• Facts: settled properties of the user's world. The database is PostgreSQL 16, the team is four people, the deploy target is Frankfurt.
• Decisions: choices made and the reason, so the assistant does not reopen a settled argument. These are the ones users are most annoyed to lose.
• Identifiers: repository names, ticket prefixes, account numbers. Short, irreplaceable, and useless if the assistant guesses instead.
• Episodic transcripts: what was said, verbatim, with timestamps. Store these if you must for audit, and keep them out of the retrieval path by default, because they are large and they contain every superseded version of everything.

The category to be suspicious of is anything transient. A user who says they are in a hurry is telling you about this minute, not about themselves, and an assistant that is still rushing three weeks later is a bug with no error message. The same goes for anything the model inferred rather than the user stated: an inference stored as a fact is indistinguishable from a fact once it is a row in a table, and it will be read back with the same confidence.

The store, and the write policy that bounds it#

The single decision that makes a memory store work is keying a row by what it is about rather than by when it was written. A store keyed by turn is an append-only log, so it grows with conversation length and holds every superseded version of every fact. A store keyed by subject has one slot per subject, so a new value overwrites the old one, growth stops at the number of distinct things the user has told you, and staleness cannot accumulate because there is nowhere for it to sit.

import re
from dataclasses import dataclass

@dataclass
class Memory:
    key: str        # what this memory is ABOUT: kind plus subject
    text: str       # the settled fact, written to stand on its own
    turn: int       # when it was last written or reconfirmed
    source: str     # the user message it came from, so it can be shown back
    seen: int = 1   # how many times the user has restated it

class MemoryStore:
    """Storage plus a write policy. One slot per subject, not one row per turn."""

    DURABLE = {"preference", "fact", "decision", "identifier"}

    def __init__(self):
        self.slots, self.log = {}, []

    def write(self, kind, subject, text, turn, source):
        if kind not in self.DURABLE:
            return self._note(turn, "rejected", f"{kind} is not durable")
        key = f"{kind}|{subject}"
        old = self.slots.get(key)
        self.slots[key] = Memory(key, text, turn, source,
                                 seen=old.seen + 1 if old and old.text == text else 1)
        if old is None:
            return self._note(turn, "new", key)
        if old.text == text:
            return self._note(turn, "reconfirmed", key)
        return self._note(turn, "superseded", f"{key}: {old.text} -> {text}")

    def _note(self, turn, action, detail):
        self.log.append((turn, action, detail))
        return action

store = MemoryStore()
transcript = [
    (1, "preference", "units", "Prefers metric units.", "give me that in metric please"),
    (2, "fact", "atlas.db", "Atlas uses a PostgreSQL 16 database.", "atlas is on postgres 16"),
    (3, "mood", "now", "Is in a hurry.", "quick, I am late for a meeting"),
    (4, "preference", "units", "Prefers metric units.", "metric again, thanks"),
    (5, "decision", "atlas.auth", "Atlas authenticates with session cookies, not JWTs.",
     "we settled on session cookies"),
    (6, "fact", "beacon.db", "Beacon uses a MySQL 8 database.", "beacon is mysql 8"),
    (7, "preference", "units", "Prefers imperial units, feet and inches.",
     "actually switch me to feet and inches"),
]
for turn, kind, subject, text, source in transcript:
    store.write(kind, subject, text, turn, source)

for turn, action, detail in store.log:
    print(f"turn {turn}  {action:12s} {detail}")
print(f"\n{len(transcript)} candidates -> {len(store.slots)} stored memories")

# turn 1  new          preference|units
# turn 2  new          fact|atlas.db
# turn 3  rejected     mood is not durable
# turn 4  reconfirmed  preference|units
# turn 5  new          decision|atlas.auth
# turn 6  new          fact|beacon.db
# turn 7  superseded   preference|units: Prefers metric units. -> Prefers imperial units, feet and inches.
#
# 7 candidates -> 4 stored memories

Four things in that class are worth naming out loud. The kind allow-list is the write policy's filter, and rejecting the mood is what stops transient state becoming permanent. The composite key is what makes superseding automatic rather than a separate reconciliation job you will never write. The seen counter distinguishes something the user has said once from something they have insisted on three times, which is the signal to use when two memories conflict and you have to pick. And the source field holds the original message, which is what you show a user who asks why the assistant thinks that, and what you show yourself when a memory turns out to be wrong.

The part this class does not do is decide what counts as a candidate in the first place. In a real system a small model reads each turn and proposes zero or more candidate writes as structured output, and the class above is what then accepts or rejects them. Keeping those two steps separate matters: extraction is fuzzy and worth a cheap model, and the policy is a handful of deterministic rules you can unit-test. A system that lets the extractor write straight to storage has no write policy, only an extractor.

The retrieval policy#

Retrieval is where the budget bites. The memories that come back are tokens in the same window as everything else, so the policy needs three things in order: a scope filter, so memories belonging to another project or another account cannot be returned at all; a relevance score with a floor, so a request that matches nothing gets nothing; and a hard budget, so the worst case is bounded rather than typical.

STOP = {"a", "an", "and", "as", "can", "do", "does", "for", "how", "i", "in", "is",
        "it", "its", "me", "my", "not", "of", "on", "or", "our", "should", "that",
        "the", "this", "to", "we", "what", "which", "with", "you"}

def terms(text):
    out = set()
    for w in re.findall(r"[a-z0-9]+", text.lower()):
        w = w[:-1] if w.endswith("s") and len(w) > 3 else w   # crude stemming
        if w not in STOP:
            out.add(w)
    return out

def retrieve(store, request, scope, budget_chars=280, floor=0.30):
    """Retrieval policy: scope, then always-on, then relevance, under one budget."""
    q = terms(request)
    always, scored = [], []
    for m in store.slots.values():
        kind, subject = m.key.split("|", 1)
        owner = subject.split(".")[0] if "." in subject else None
        if owner is not None and owner != scope:
            continue                                   # belongs to another project
        if kind == "preference":
            always.append(m)                           # few, small, always relevant
            continue
        t = terms(m.text)
        scored.append((len(q & t) / len(q) if q else 0.0, m))

    scored.sort(key=lambda row: -row[0])
    kept, used = list(always), sum(len(m.text) for m in always)
    for score, m in scored:
        if score < floor or used + len(m.text) > budget_chars:
            break
        kept.append(m)
        used += len(m.text)
    return kept, scored

for request, scope in [("what database does atlas use?", "atlas"),
                       ("how tall is the Eiffel tower?", "atlas"),
                       ("what database does beacon use?", "beacon")]:
    kept, scored = retrieve(store, request, scope)
    print(f"\n{request!r}   scope={scope}")
    for score, m in scored:
        print(f"   {'IN ' if m in kept else 'out'} relevance {score:.2f}  {m.text}")
    print(f"   sent: {len(kept)} memories, {sum(len(m.text) for m in kept)} chars")

#
# 'what database does atlas use?'   scope=atlas
#    IN  relevance 0.75  Atlas uses a PostgreSQL 16 database.
#    out relevance 0.25  Atlas authenticates with session cookies, not JWTs.
#    sent: 2 memories, 76 chars
#
# 'how tall is the Eiffel tower?'   scope=atlas
#    out relevance 0.00  Atlas uses a PostgreSQL 16 database.
#    out relevance 0.00  Atlas authenticates with session cookies, not JWTs.
#    sent: 1 memories, 40 chars
#
# 'what database does beacon use?'   scope=beacon
#    IN  relevance 0.75  Beacon uses a MySQL 8 database.
#    sent: 2 memories, 71 chars

The middle case is the one to look at. A question about the Eiffel tower returns one memory, the always-on preference, and nothing else, because the floor rejected everything below 0.30. That empty result is correct behaviour and it is the behaviour most implementations get wrong, because a top-k search always returns k things however irrelevant they are. Top-k with no floor is how an assistant ends up being told about your database while you are asking about a tower, and the model has no way to know the context it was given is noise.

Splitting preferences out from retrieved facts is the other decision worth defending. Preferences are a handful of rows, they are short, and they are relevant to nearly every request, so paying to retrieve them is more expensive than paying to send them. Facts are the opposite: many, longer, and relevant to a few requests each. A single ranked list treats both the same and so either floods the window with preferences or drops them on the requests where they mattered. The budget itself belongs in the same accounting as everything else in the window; the window section in this topic has the audit that shows where the tokens are actually going.

The three ways memory goes wrong#

Stale facts that contradict newer ones, unbounded growth, and confidently retrieving the wrong memory. They sound like three problems and they are close to one: all three follow from a store that records when something was said instead of what it was about.

import random

rng = random.Random(0)
SUBJECTS = [f"subject{i:02d}" for i in range(40)]

append_only, slotted = [], {}
for turn in range(1, 2001):
    s = rng.choice(SUBJECTS)
    text = f"{s} is set to value{rng.randrange(5)}."
    append_only.append((turn, s, text))
    slotted[s] = (turn, s, text)

print(f"2000 turns over {len(SUBJECTS)} subjects")
print(f"  append-only rows {len(append_only):5d}   {sum(len(t) for _, _, t in append_only):6d} chars")
print(f"  one slot each    {len(slotted):5d}   {sum(len(t) for _, _, t in slotted.values()):6d} chars")

versions = [t for _, s, t in append_only if s == SUBJECTS[0]]
print(f"\neverything the store knows about {SUBJECTS[0]}")
print(f"  append-only: {len(versions)} memories, {len(set(versions))} mutually exclusive")
print(f"  one slot:    1 memory, {slotted[SUBJECTS[0]][2]}")

# The third failure: a fact whose subject was never recorded.
flat = MemoryStore()
flat.write("fact", "db", "Uses a PostgreSQL 16 database.", 1, "atlas is on postgres 16")
kept, scored = retrieve(flat, "what database does beacon use?", scope="beacon")
print("\nasked about Beacon, given a memory that never said which project it was about")
for score, m in scored:
    print(f"  relevance {score:.2f}  {m.text}   -> {'sent' if m in kept else 'held back'}")

# 2000 turns over 40 subjects
#   append-only rows  2000    54000 chars
#   one slot each       40     1080 chars
#
# everything the store knows about subject00
#   append-only: 51 memories, 5 mutually exclusive
#   one slot:    1 memory, subject00 is set to value3.
#
# asked about Beacon, given a memory that never said which project it was about
#   relevance 0.50  Uses a PostgreSQL 16 database.   -> sent

Staleness first. The append-only store holds fifty-one memories about subject00 and five of them cannot all be true. Retrieval scores by relevance, so all five score identically, and whichever ones fit the budget go into the prompt. The model is then given a contradiction with no way to resolve it, and what it does with that is not predictable. If you have to keep history, give each row a validity flag and retrieve only the current one, so the audit trail lives in storage rather than in the retrieval path.

Growth second. Fifty times the rows for the same knowledge, and the ratio keeps rising because the subject count is fixed by the user's world while the turn count is not. Unbounded growth rarely shows up as a storage bill. It shows up as retrieval getting slower and less precise, because every query now ranks against thousands of near-duplicate rows and the top results are all versions of the same thing.

Confident wrong retrieval third, and it is the one that damages trust. The memory in the last block is a true sentence: something does use PostgreSQL 16. It was written without recording which project it was about, so it matches a question about a different project at 0.50 and is sent. Nothing in the pipeline errors, the model has no reason to doubt it, and the user gets a specific, wrong, fluent answer. The fix is in the write policy rather than the retrieval one: a fact whose subject you cannot name is a fact you should not store.

Show it, and let people delete it#

Everything above assumes you can inspect the store, which is why the source field exists on the record. A memory view is a list of rows with what is remembered, when it was written, the message it came from, and a delete button, and it is perhaps an afternoon of work. Without it a wrong memory is undiagnosable from the outside and unfixable from the inside: the user sees an assistant that is oddly wrong about them and has no way to say why, and you see a support ticket that reproduces for one account and no other.