Context windows
Big is not the same as usable.
A model has no memory between calls. Everything it knows about your conversation has to be sent to it again, every single time, as one block of text. The context window is the size limit on that block — the most text the model can look at in one go, counted in tokens rather than words.
Two things follow from it, and both cost money. You pay for every token you send, on every call, so a conversation that keeps growing keeps getting more expensive. And when the block gets too big the call simply fails, usually in production and usually on your longest and most important request.
It sits on top of tokenization, which is what turns your text into the tokens being counted. Almost everything else in AI engineering is a way of deciding what deserves a place in this limited space: retrieval finds the few documents worth sending, context management decides what to throw away as a conversation grows, and prompt caching stops you paying twice for the part that never changes.
The window is a budget you spend on every call, not a container you fill once.
In more detailContext limits have grown enormously, but attention quality is not uniform across a long window: material in the middle is recalled less reliably than material at either end. Cost also scales with everything you send, so filling a large window because you can is usually both worse and more expensive than curating what goes in.
Learn first
The window as a budget
By the end: You can predict whether a request fits and decide what to cut when it does not.
- What counts against the window
- Leaving room for the reply
- Uneven recall across a long window
- Deciding what to drop
Every call you make sends the whole conversation again, and you are charged for all of it. That makes the window an allowance you spend rather than a container you fill: this section is about working out what is actually in it, what it costs, and what to drop first when a request will not fit. The part people forget is that the reply needs room too, and a call that leaves none fails at the worst moment.
What the window is#
The context window is the maximum number of tokens the model can have in front of it during one call — prompt and reply together. It is not memory and it is not storage. Nothing you sent last time is there this time; the model starts every call from nothing and reads whatever you hand it. A conversation only exists because your code resends it, and the window is the ceiling on how much of it you can resend.
The recognition signal is any sentence containing the words remembers, earlier, or the conversation so far. If a feature depends on something the model was told before, some piece of your code is deciding whether that thing is worth its tokens on this call. If no code is making that decision, the decision is being made by whatever order your message list happens to be in.
Everything counts#
The window covers the system prompt, tool definitions, every message in the history, every tool result, any retrieved documents, and the response being generated. People routinely forget the last two.
SYSTEM = "You are a support assistant. " * 60
TOOL_DEFINITIONS = '{"name": "lookup_order", "input_schema": {}} ' * 40
def count_tokens(text):
"""Stand-in. Use your provider's counting endpoint before you ship;
four characters to a token is close enough to decide what to cut."""
return len(text) // 4
def fits(messages, documents, reply_room=2000, limit=128_000):
used = (
count_tokens(SYSTEM)
+ count_tokens(TOOL_DEFINITIONS)
+ sum(count_tokens(m) for m in messages)
+ sum(count_tokens(d) for d in documents)
)
return used + reply_room <= limit, used
messages = ["a turn of conversation. " * 40] * 30
documents = ["a retrieved passage. " * 300] * 12
ok, used = fits(messages, documents)
print(f"used {used:,} + 2,000 reply room = {used + 2000:,} against 128,000 -> fits: {ok}")
print(f" system {count_tokens(SYSTEM):>7,}")
print(f" tools {count_tokens(TOOL_DEFINITIONS):>7,}")
print(f" history{sum(count_tokens(m) for m in messages):>7,}")
print(f" docs {sum(count_tokens(d) for d in documents):>7,}")
many = ["a retrieved passage. " * 300] * 76
ok, used = fits(messages, many)
print(f"\nwith 76 documents: {used:,} used, {used + 2000:,} with reply room -> fits: {ok}")
ok_no_room, _ = fits(messages, many, reply_room=0)
print(f"the same call, forgetting the reply, says fits: {ok_no_room}")
print("It will be accepted, and the answer will truncate after a few hundred tokens.")
# used 26,985 + 2,000 reply room = 28,985 against 128,000 -> fits: True
# system 435
# tools 450
# history 7,200
# docs 18,900
#
# with 76 documents: 127,785 used, 129,785 with reply room -> fits: False
# the same call, forgetting the reply, says fits: True
# It will be accepted, and the answer will truncate after a few hundred tokens.The arithmetic of what fits#
Intuition about what fits in two hundred thousand tokens is bad, in both directions. People assume a long contract is hopeless and it is not, and assume a mailbox will be fine and it is nowhere close. English prose runs at roughly three quarters of a word per token, which is enough to settle the question before you write any code.
def tokens_from_words(words):
"""English prose runs about 0.75 words per token. Count properly before you
ship; this is accurate enough to decide whether something is even close."""
return round(words / 0.75)
LIMIT = 200_000
SYSTEM, TOOLS, REPLY_ROOM = 1_800, 3_400, 4_000
items = [
("a 2-page memo", 2 * 500),
("a 40-page vendor contract", 40 * 450),
("a 300-page annual report", 300 * 600),
("a 90-minute call transcript", 90 * 140),
("this year's support email", 4_000 * 180),
]
print(f"{'artefact':<28}{'words':>10}{'tokens':>10}{'% of a 200k window':>21}")
for name, words in items:
t = tokens_from_words(words)
print(f"{name:<28}{words:>10,}{t:>10,}{t/LIMIT:>20.1%}")
usable = LIMIT - SYSTEM - TOOLS - REPLY_ROOM
print(f"\nwindow {LIMIT:>9,}")
print(f"system prompt -{SYSTEM:>9,}")
print(f"tool definitions -{TOOLS:>9,}")
print(f"room for the reply -{REPLY_ROOM:>9,}")
print(f"left for content {usable:>9,} ({usable/LIMIT:.1%})")
contract = tokens_from_words(40 * 450)
print(f"\nThe contract at {contract:,} tokens fits {usable // contract} times over.")
report = tokens_from_words(300 * 600)
print(f"The annual report at {report:,} does not fit at all: over by {report - usable:,}.")
# artefact words tokens % of a 200k window
# a 2-page memo 1,000 1,333 0.7%
# a 40-page vendor contract 18,000 24,000 12.0%
# a 300-page annual report 180,000 240,000 120.0%
# a 90-minute call transcript 12,600 16,800 8.4%
# this year's support email 720,000 960,000 480.0%
#
# window 200,000
# system prompt - 1,800
# tool definitions - 3,400
# room for the reply - 4,000
# left for content 190,800 (95.4%)
#
# The contract at 24,000 tokens fits 7 times over.
# The annual report at 240,000 does not fit at all: over by 49,200.A conversation is quadratic#
The window fills up gradually and the bill does not. Each turn resends everything before it, so total tokens billed grows with the square of the turn count while the conversation itself grows linearly. This is the single most surprising line on a first invoice.
SYSTEM, PER_TURN = 1_800, 600 # fixed preamble; user message plus reply per turn
IN_PER_MTOK = 3.00
PREFILL_TOK_PER_S = 8_000 # measure yours; prefill time is roughly linear
print(f"{'turn':>5}{'window':>10}{'billed so far':>15}{'$ so far':>10}{'prefill':>10}")
billed = 0
for turn in range(1, 41):
window = SYSTEM + PER_TURN * turn
billed += window
if turn in (1, 5, 10, 20, 30, 40):
print(f"{turn:>5}{window:>10,}{billed:>15,}{billed/1e6*IN_PER_MTOK:>10.3f}"
f"{window/PREFILL_TOK_PER_S:>9.2f}s")
final = SYSTEM + PER_TURN * 40
print(f"\nThe conversation ends at {final:,} tokens. You were billed for {billed:,}.")
print(f"That is {billed/final:.1f}x the size of the thing itself, "
f"because every turn resends it.")
print(f"Turn 40 costs {(SYSTEM+PER_TURN*40)/(SYSTEM+PER_TURN):.1f}x turn 1 and its prefill "
f"is {(SYSTEM+PER_TURN*40)/(SYSTEM+PER_TURN):.1f}x longer, before the model writes "
f"a word.")
# turn window billed so far $ so far prefill
# 1 2,400 2,400 0.007 0.30s
# 5 4,800 18,000 0.054 0.60s
# 10 7,800 51,000 0.153 0.97s
# 20 13,800 162,000 0.486 1.73s
# 30 19,800 333,000 0.999 2.48s
# 40 25,800 564,000 1.692 3.23s
#
# The conversation ends at 25,800 tokens. You were billed for 564,000.
# That is 21.9x the size of the thing itself, because every turn resends it.
# Turn 40 costs 10.8x turn 1 and its prefill is 10.8x longer, before the model writes a word.Cost and latency are the same curve here, which is why a long chat feels sluggish as well as costing more: the prefill at turn forty is 3.2 seconds before a single token comes back. Both are fixed by the same two moves — cache the stable prefix so the re-read is nearly free, and compact the history so it stops growing — and prompt caching only works if the history is appended to rather than rewritten, which is a constraint on how you compact.
Recall is not uniform#
A large window does not mean everything in it is equally usable. Material near the beginning and the end is recalled more reliably than material in the middle. This is well enough established to design around.
The published result is a U-shaped curve: accuracy at finding a fact is highest when it sits at the start or the end of the context, sags in the middle, and the sag deepens as the context gets longer. It is a retrieval-within-the-prompt effect rather than a limit on the window, which is why it shows up most on tasks where one specific passage carries the answer and barely at all on summarisation, where everything matters a little. The practical reading is that position is a variable you control and should be setting deliberately.
- Put the most important content near the start or the end of what you send.
- Do not bury a critical instruction in the middle of a long document.
- Fewer, better documents usually beat more documents, for accuracy as well as cost.
Rank order is the worst arrangement available, because it puts your fourth-best passage exactly where recall is weakest. Reordering costs nothing.
def sandwich(passages):
"""Rank order buries the best of the middle passages exactly where recall is
weakest. This puts the strongest at the two ends and the weakest in between."""
ranked = sorted(passages, key=lambda p: -p[1])
front, back = [], []
for i, p in enumerate(ranked):
(front if i % 2 == 0 else back).append(p)
return front + back[::-1]
passages = [(f"doc{i}", round(0.91 - 0.05 * i, 2)) for i in range(10)]
print("rank order :", [p[0] for p in passages])
print(" ", [p[1] for p in passages])
ordered = sandwich(passages)
print("sandwiched :", [p[0] for p in ordered])
print(" ", [p[1] for p in ordered])
for label, order in (("rank order", passages), ("sandwiched", ordered)):
middle = [p[1] for p in order[3:7]]
print(f"\nthe four weakest-recalled positions hold, {label}: {middle}")
print(f" best passage stranded there: {max(middle)}")
# rank order : ['doc0', 'doc1', 'doc2', 'doc3', 'doc4', 'doc5', 'doc6', 'doc7', 'doc8', 'doc9']
# [0.91, 0.86, 0.81, 0.76, 0.71, 0.66, 0.61, 0.56, 0.51, 0.46]
# sandwiched : ['doc0', 'doc2', 'doc4', 'doc6', 'doc8', 'doc9', 'doc7', 'doc5', 'doc3', 'doc1']
# [0.91, 0.81, 0.71, 0.61, 0.51, 0.46, 0.56, 0.66, 0.76, 0.86]
#
# the four weakest-recalled positions hold, rank order: [0.76, 0.71, 0.66, 0.61]
# best passage stranded there: 0.76
#
# the four weakest-recalled positions hold, sandwiched: [0.61, 0.51, 0.46, 0.56]
# best passage stranded there: 0.61What to cut, in order#
- 1.Old tool results. Rarely needed later and usually the largest single block.
- 2.Middle conversation turns, summarised into a short note.
- 3.Low-scoring retrieved documents. If your retrieval returns ten and the top three answer the question, the rest are cost.
- 4.Verbose instructions that the model already follows. Long system prompts accumulate lines nobody has tested.
Never cut the task
The original request, any user corrections, and constraints already agreed must survive every trim. An agent that loses the task keeps working confidently on the wrong problem, and nothing errors.
What to skip#
Skip building a compaction strategy for a feature whose conversations are six turns long; the arithmetic above says it is not worth the code. Skip token counting on the hot path if a character estimate is within ten per cent and you have thirty per cent headroom, and spend the call on the one place it matters, which is the boundary check before sending. And skip the instinct that a bigger window removes the problem: it moves the failure from an error you would have seen to a cost and a recall curve you will not.
Where people go wrong
Checking only the input size. If input nearly fills the window, generation has almost no room and the response truncates immediately.
Build this: Write a prompt assembler that takes a token budget and drops the lowest-value content until it fits, then prove it never drops the original task.
Open this section on its own pageWorking with inputs bigger than you want to send
By the end: You can handle a long document or conversation by choosing between retrieval, staged summaries and a long prompt, with reasons.
- Count tokens before you send
- A budget for each part of the prompt
- Retrieval, map-reduce, or one long call
- Where to put what matters most
- Summaries that keep what later turns need
Sooner or later what you want to send does not fit, or fits and costs more than it is worth. This section is about that decision. Count the real tokens before sending rather than learning the limit from an error, give each part of the prompt a budget so you know what gives way when something grows, then choose between retrieval, staged summaries and one long call, each of which is right in different circumstances. It also covers where in a long prompt to put what matters most, because recall across a window is not uniform.
Count before you send#
Guessing token counts from characters is fine for estimates and wrong for limits. Most providers offer a token counting endpoint or library. Count the real prompt before sending, so you can decide what to cut instead of discovering it from an error.
The estimate goes wrong in one direction and always on the inputs you care about. Four characters to a token holds for English prose; it does not hold for source code, which is dense in punctuation, for JSON, where every brace and quote is its own token, for languages that do not use the Latin alphabet, which can run several times worse, or for anything base64 encoded, which tokenises close to one token per two or three characters. A retrieval corpus of API documentation can be forty per cent above the estimate, and forty per cent is the difference between comfortable and rejected.
Give every part a budget#
Window 200k tokens (example)
system prompt and tools 6k fixed
retrieved documents 40k cut lowest-ranked first
conversation history 20k summarise oldest turns
user message 4k reject beyond this, with a clear message
reserved for the reply 8k
headroom the rest, because long prompts cost and slowBudgets turn a vague worry into a rule each part must follow. When something grows, you know what gives way.
Two details make the difference between a budget that works and one that reads well. Each line needs a named policy, not just a number, because the interesting question is what happens at the limit — the user message line above rejects with a message, the documents line drops the lowest-ranked, and those are different behaviours that have to be written somewhere. And the budget should be enforced in one function that every code path goes through, because the failure people actually hit is a second, less-travelled path that assembles the prompt its own way and has no budget at all.
Three ways to handle a long document#
- Retrieval: the question needs one part of the document. Find that part and send only it. Cheapest and usually most accurate.
- Map-reduce: the question needs all of it, such as a summary or 'list every obligation'. Process each section separately, then combine the results.
- One long call: the document fits, the question needs connections across it, and the cost is acceptable. Simplest to build, most expensive per question.
The question decides, not the document size. A question with one answer in one place is a retrieval question however long the document is. A question of the form every, all or how many is a map-reduce question, because retrieval will confidently return the top five of the eleven obligations and nothing will indicate that six are missing. A question about how one part relates to another is the case for a single long call, because that is the only arrangement in which both parts are in front of the model at the same time.
SENT = []
def ask(prompt, reply_tokens=400):
"""Stand-in for one model call, so the accounting below is real.
In your code this is client.chat([{"role": "user", "content": prompt}])."""
SENT.append((len(prompt) // 4, reply_tokens))
return f"[summary of {len(prompt) // 4} tokens]"
def summarise_long(sections):
partials = [ask("Summarise, keeping every figure and name:\n" + s) for s in sections]
return ask("Combine these section summaries into one, "
"keeping section references:\n" + "\n\n".join(partials))
document = "word " * 240_000 # about 300,000 tokens
sections = [document[i:i + 80_000] for i in range(0, len(document), 80_000)]
summarise_long(sections)
sent_in = sum(a for a, _ in SENT)
sent_out = sum(b for _, b in SENT)
biggest = max(a for a, _ in SENT)
one_call = len(document) // 4
print(f"{len(sections)} sections, {len(SENT)} model calls")
print(f"largest single call {biggest:>9,} input tokens")
print(f"total input billed {sent_in:>9,}")
print(f"total output billed {sent_out:>9,}")
print(f"one long call {one_call:>9,} input tokens, in a single request")
print(f"\nMap-reduce bills {sent_in/one_call:.2f}x the input of the long call and "
f"{sent_out/400:.0f}x the output,")
print(f"but no request exceeds {biggest:,} tokens, and the {len(sections)} map calls "
f"run in parallel.")
print("The saving is not money. It is that the job is possible at all.")
# 15 sections, 16 model calls
# largest single call 20,010 input tokens
# total input billed 300,268
# total output billed 6,400
# one long call 300,000 input tokens, in a single request
#
# Map-reduce bills 1.00x the input of the long call and 16x the output,
# but no request exceeds 20,010 tokens, and the 15 map calls run in parallel.
# The saving is not money. It is that the job is possible at all.The thing people expect from map-reduce is a saving, and there is not one: the same text goes through the model either way, plus the partial summaries on top. What it buys is that every request fits, that the fifteen map calls run at the same time so wall-clock latency is one section rather than fifteen, and that a failure costs you one section instead of the whole job. What it costs is the information that crossed a section boundary, which is why a clause on page four modifying a clause on page ninety is the case map-reduce quietly gets wrong.
Longer context or retrieval: the arithmetic#
This is the design question that actually comes up, and it is usually answered by taste. It does not have to be. There are three options — send everything every time, send everything but cache the prefix, or retrieve a slice — and the inputs that decide between them are the size of the corpus, how many questions you will ask it, and how long it takes to build the retrieval pipeline.
IN, OUT = 3.00, 15.00 # dollars per million tokens. Substitute yours.
EMBED_PER_MTOK = 0.02
CACHE_WRITE, CACHE_READ = 1.25, 0.10 # multipliers on the input rate
DOC, RETRIEVED = 180_000, 6_000
PROMPT, REPLY = 400, 500
BUILD_HOURS, HOURLY = 24, 90 # what the retrieval pipeline costs to build and wire up
def q_cost(context, mult=1.0):
return (context * mult + PROMPT) / 1e6 * IN + REPLY / 1e6 * OUT
long_q = q_cost(DOC)
short_q = q_cost(RETRIEVED)
cached_q = q_cost(DOC, CACHE_READ)
index_once = DOC / 1e6 * EMBED_PER_MTOK
build_once = BUILD_HOURS * HOURLY
print(f"whole document every time " "$" f"{long_q:>8.4f} a question")
print(f"whole document, cached prefix " "$" f"{cached_q:>8.4f} a question "
"(plus " "$" f"{q_cost(DOC, CACHE_WRITE) - long_q:.4f} on the first, "
"and only while the entry lives)")
print(f"retrieve 6k of it " "$" f"{short_q:>8.4f} a question")
print(f"\n{'questions':>10}{'long':>12}{'cached':>12}{'retrieval':>12}{'retrieval+build':>17}")
for n in (1, 100, 1_000, 10_000, 100_000):
a, b, c = n * long_q, n * cached_q, index_once + n * short_q
print(f"{n:>10,}{a:>12,.0f}{b:>12,.0f}{c:>12,.0f}{c + build_once:>17,.0f}")
n_tokens = build_once / (long_q - short_q)
n_cached = build_once / (cached_q - short_q)
print(f"\nOn tokens alone retrieval wins from the first question. Including the "
f"{BUILD_HOURS} hours to build it at " "$" f"{HOURLY}/h, it wins from question "
f"{n_tokens:,.0f}.")
print(f"Against a cached prefix it needs {n_cached:,.0f} questions, which most internal "
f"tools never reach.")
# whole document every time $ 0.5487 a question
# whole document, cached prefix $ 0.0627 a question (plus $0.1350 on the first, and only while the entry lives)
# retrieve 6k of it $ 0.0267 a question
#
# questions long cached retrieval retrieval+build
# 1 1 0 0 2,160
# 100 55 6 3 2,163
# 1,000 549 63 27 2,187
# 10,000 5,487 627 267 2,427
# 100,000 54,870 6,270 2,670 4,830
#
# On tokens alone retrieval wins from the first question. Including the 24 hours to build it at $90/h, it wins from question 4,138.
# Against a cached prefix it needs 60,000 questions, which most internal tools never reach.Read the table as a decision rather than a cost comparison, because the token numbers alone say build retrieval and the token numbers alone are misleading. A corpus small enough to cache and fewer than a few thousand questions: cache the prefix and write no retrieval code, and note that this is the option nobody proposes in a design review. Many questions, or a corpus too large to sit in one window, or one that changes constantly so no cache entry survives: build retrieval, and the four thousand questions it takes to pay for itself will arrive in a week. Sending the whole document uncached every time is the one arrangement that is never right at volume, and it is the one that gets shipped, because it is what the prototype did.
Two things the arithmetic does not capture and you should say out loud in an interview. Cache entries expire in minutes, so the cached column only applies to bursts of questions, not to twenty a day spread over office hours. And retrieval that returns the wrong passage is wrong for free, whereas a long context that contains the answer is at worst expensive — which is why the accuracy comparison, not the cost one, is what an eval has to settle.
Position matters#
Models tend to use information at the start and end of a long context more reliably than information buried in the middle. Put instructions and the question where they are hard to miss, and put the most relevant retrieved passages first or last rather than in the middle of a long list.
Summaries that keep what matters#
When a conversation is compressed, a generic summary loses the details later turns depend on: ids, numbers, constraints the user stated, things already tried. Ask for those explicitly, or better, keep them in a structured record next to the summary.
Summary of turns 1 to 30:
Goal: move Jack's team from the monthly to the annual plan
Facts: account 4471, 12 seats, renewal date 1 March
Decided: keep current seat count
Tried and failed: coupon SPRING (expired)
Open: whether annual includes the support add-onTried and failed is the line that earns its place. Without it a compacted agent retries the thing that did not work, sometimes repeatedly, and the loop looks like a reasoning failure when it is a memory one. Keep that record as structured fields your code writes and reads, not as prose the model regenerates each time it compacts, because a summary of a summary loses a little more on every pass and nothing tells you which pass dropped the account number.
What to skip#
Skip retrieval for a corpus that fits in the window and gets asked a few dozen questions a day; cache it instead and spend the fortnight elsewhere. Skip map-reduce when the answer lives in one place, since it pays for the whole document to answer a question about one paragraph. And skip tuning chunk sizes before you have measured whether retrieval is even the failing stage — that is the debugging bisect from the debugging topic, applied here.
Where people go wrong
Letting the prompt grow until it hits the limit and then cutting whatever is easiest. The cut should be decided in advance, by budget.
Build this: Take a document of fifty pages or more. Answer the same ten questions three ways: retrieval, map-reduce and one long call. Record accuracy, cost and latency for each and write down which you would ship.
Open this section on its own pageWhere you use this
A topic is easier to care about when you can see what it is for. These projects lean on it at a named stage.
- Stage 5, Retrieve widely, then rank precisely: Why the answer is not to send forty chunks and let the model sort it out.
Now actually do it
Reading a topic is not learning it. These are the hours that count, and each one tells you when you are finished.
1Build a prompt assembler with a budget
4hAssemble a prompt from system, history and documents inside a token budget, dropping lowest-value content until it fits, and always reserving room for the reply.
Done when: It never drops the original task, and it never fills the window leaving no room to generate.
4 hours of practice
Words used here
1 term this topic teaches, each with a one-line definition if one of them stopped you.
- Context window
- The maximum number of tokens a model can attend to in one call — system prompt, retrieved material, conversation history and the reply being generated, all counted together.
Interview questions
6 questions, grouped by the section that teaches the answer. An answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
The window as a budget
What counts against the context window?
Everything sent plus everything generated: system prompt, tool definitions, all messages, all tool results, retrieved documents and the reply itself. Forgetting to reserve room for the reply is a common cause of failures.
Is recall uniform across a long context window?
No. Content near the start and the end is recalled more reliably than content in the middle. So position matters, and filling a large window because it exists is worse than curating what goes into it.
Why does a long conversation get expensive faster than it gets long?
Because every turn resends everything before it. The conversation grows linearly and the tokens billed grow with the square of the turn count: forty turns of a conversation that ends at 25,800 tokens bills about 564,000, roughly 22 times its own final size. Prefill latency follows the same curve, so turn 40 is also about ten times slower to start than turn 1. Prompt caching and compaction fix both, and caching only works if history is appended rather than rewritten.
What gets marked up. Strong answers name the quadratic and connect it to latency as well as cost.
A request does not fit. What do you cut first?
Old tool results, then middle conversation turns summarised, then low-scoring retrieved documents, then untested instructions. Never the original task, user corrections or agreed constraints.
Working with inputs bigger than you want to send
A model supports a very long context. Why not always send everything?
Cost and latency grow with every token, recall is weaker for material in the middle of a long context, and irrelevant text can distract the model. Sending only what is relevant is usually cheaper, faster and more accurate.
Then they push further
When is sending the whole document the right call?
When the question needs connections across the whole thing, the document fits comfortably, and the volume is low enough that cost does not matter, for example a one-off review of a contract.
How would you prove which approach is better for your case?
Build a small eval with questions of each kind, run retrieval, map-reduce and the long call on it, and compare accuracy, cost and latency side by side.
What gets marked up. Strong answers name cost, latency and uneven recall, and choose by the kind of question. Treating a bigger window as a replacement for retrieval is marked down.
What would you keep when compressing a long conversation?
The goal, ids and numbers, constraints the user stated, decisions made and attempts that failed, ideally in a structured record, plus the most recent turns word for word.
Resources
Free to watch and read, picked for this topic.
Videos
For the window as a budget
For working with inputs bigger than you want to send
Full courses: the whole topic, start to finish
Stop 13 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.