The model has no memory
By the end: You can explain why conversations work the way they do, and build one correctly.
- What stateless means here
- Proving it in two calls
- Building a conversation by hand
- Why cost grows faster than the conversation does
New to the messages api? Using a language model means sending it a request over the internet and getting a reply back. The request is a list of messages, each labelled with who said it — standing instructions, the person, or the model's own earlier replies. The response is one more message. That exchange is the entire interface, and the model keeps nothing afterwards: it has no memory of the request you sent a second ago. Why it matters
Chat products feel as though they remember you. The thing underneath remembers nothing at all, and this section is about what follows from that. You will prove it to yourself, then build a conversation the way every chat product does, by resending the entire history on every turn. The consequence that catches people out lands on the bill, because the twentieth turn is paying for the first nineteen all over again.
Every call is the first call#
The model remembers nothing between requests. Each call is a separate HTTP request that carries everything the model will know. Nothing is stored on the other end.
This is easy to accept and hard to believe, because chat products feel like they remember. Prove it to yourself in two calls.
ask("My name is Jack.")
# "Nice to meet you, Jack!"
ask("What is my name?")
# "I don't have access to your name."So what is a conversation?#
Something you build. To make turn five work, you resend turns one to four along with the new question. The illusion of memory is entirely on your side of the wire.
history = []
def ask(text):
history.append({"role": "user", "content": text})
reply = client.chat(history) # send ALL of it
history.append({"role": "assistant", "content": reply.text})
return reply.textAppend the model's replies too
Forgetting the second append is the most common mistake here. The model then sees a run of user messages with no answers in between, and gets confused about what it already said. The symptom is oddly incoherent later turns.
Why this gets expensive#
Each turn resends everything before it. Turn one sends one message, turn ten sends nineteen. Total cost across a conversation grows with the square of its length, not in proportion to it.
Turn 1: 7 input tokens
Turn 2: 33 input tokens
Turn 3: 48 input tokensThere is a second surprise in those numbers. Model replies are usually longer than user questions, so most of what you are paying to resend is the model's own earlier output.
Every technique you will meet later, prompt caching, summarising old turns, retrieval instead of history, is an answer to this one problem.
Where people go wrong
Not appending the assistant's replies to the history. The model receives several user turns in a row with no responses between them, and later answers become subtly wrong about what was already said.
If this comes up in an interview
Are LLM APIs stateful or stateless?
Stateless. The model retains nothing between calls. Conversation is created by the caller resending the full history each turn. Some products offer a session id, but underneath they are storing and resending the history for you.
Why does a long conversation get expensive faster than you would expect?
Every turn resends all previous turns, so total tokens grow with the square of the number of turns rather than linearly. Most of that resent text is the model's own previous output, which is typically longer than the user's messages.
Then they push further
Roughly how much faster?
If each turn adds a similar amount, total input over n turns grows with the square of n, because every turn resends everything before it. Twice as many turns costs roughly four times as much input.
What would you do about it?
Cache the stable prefix, trim or summarise older turns, and drop bulky tool results once they are no longer needed.
What gets marked up. Strong answers explain that the API is stateless so the whole history is sent every turn, and can reason about the growth.
How would you keep a long-running chat affordable?
Prompt caching on the stable prefix first, since it is free quality-wise. Then summarise or drop older turns, keeping a short running summary. Then retrieve only relevant history instead of carrying all of it. Increasing the context window is the last resort, not the first.
Build this: Build a conversation loop that prints the input token count each turn. Run it for five turns with short questions and watch the count climb. Then remove the assistant appends and see how the answers degrade.
Watch it explained
This is one section of The Messages API, 6 hours across 3 sections. Read it in full to get the whole topic in order.
Stop 4 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.