Asked about The model has no memory
How would you keep a long-running chat affordable?
An answer that works
Prompt caching on the stable prefix first, since it is free quality-wise. Then summarise or drop older turns, keeping a short running summary. Then retrieve only relevant history instead of carrying all of it. Increasing the context window is the last resort, not the first.
Reading an answer is not being able to give one. The section that teaches this is The model has no memory, 2 hours of The Messages API.
Learn this properlyAlso asked about The Messages API
What are the roles in a chat request and what is each for?What is stop_reason and why check it?Does max_tokens make the model give shorter answers?Are LLM APIs stateful or stateless?Why does a long conversation get expensive faster than you would expect?What does temperature do?Is temperature 0 deterministic?How do you make a model give shorter answers?
All 10 questions on The Messages API