The window as a budget
By the end: You can predict whether a request fits and decide what to cut when it does not.
- What counts against the window
- Leaving room for the reply
- Uneven recall across a long window
- Deciding what to drop
New to context windows? A model has no memory between calls. Everything it knows about your conversation has to be sent to it again, every single time, as one block of text. The context window is the size limit on that block — the most text the model can look at in one go, counted in tokens rather than words. Why it matters
Every call you make sends the whole conversation again, and you are charged for all of it. That makes the window an allowance you spend rather than a container you fill: this section is about working out what is actually in it, what it costs, and what to drop first when a request will not fit. The part people forget is that the reply needs room too, and a call that leaves none fails at the worst moment.
What the window is#
The context window is the maximum number of tokens the model can have in front of it during one call — prompt and reply together. It is not memory and it is not storage. Nothing you sent last time is there this time; the model starts every call from nothing and reads whatever you hand it. A conversation only exists because your code resends it, and the window is the ceiling on how much of it you can resend.
The recognition signal is any sentence containing the words remembers, earlier, or the conversation so far. If a feature depends on something the model was told before, some piece of your code is deciding whether that thing is worth its tokens on this call. If no code is making that decision, the decision is being made by whatever order your message list happens to be in.
Everything counts#
The window covers the system prompt, tool definitions, every message in the history, every tool result, any retrieved documents, and the response being generated. People routinely forget the last two.
SYSTEM = "You are a support assistant. " * 60
TOOL_DEFINITIONS = '{"name": "lookup_order", "input_schema": {}} ' * 40
def count_tokens(text):
"""Stand-in. Use your provider's counting endpoint before you ship;
four characters to a token is close enough to decide what to cut."""
return len(text) // 4
def fits(messages, documents, reply_room=2000, limit=128_000):
used = (
count_tokens(SYSTEM)
+ count_tokens(TOOL_DEFINITIONS)
+ sum(count_tokens(m) for m in messages)
+ sum(count_tokens(d) for d in documents)
)
return used + reply_room <= limit, used
messages = ["a turn of conversation. " * 40] * 30
documents = ["a retrieved passage. " * 300] * 12
ok, used = fits(messages, documents)
print(f"used {used:,} + 2,000 reply room = {used + 2000:,} against 128,000 -> fits: {ok}")
print(f" system {count_tokens(SYSTEM):>7,}")
print(f" tools {count_tokens(TOOL_DEFINITIONS):>7,}")
print(f" history{sum(count_tokens(m) for m in messages):>7,}")
print(f" docs {sum(count_tokens(d) for d in documents):>7,}")
many = ["a retrieved passage. " * 300] * 76
ok, used = fits(messages, many)
print(f"\nwith 76 documents: {used:,} used, {used + 2000:,} with reply room -> fits: {ok}")
ok_no_room, _ = fits(messages, many, reply_room=0)
print(f"the same call, forgetting the reply, says fits: {ok_no_room}")
print("It will be accepted, and the answer will truncate after a few hundred tokens.")
# used 26,985 + 2,000 reply room = 28,985 against 128,000 -> fits: True
# system 435
# tools 450
# history 7,200
# docs 18,900
#
# with 76 documents: 127,785 used, 129,785 with reply room -> fits: False
# the same call, forgetting the reply, says fits: True
# It will be accepted, and the answer will truncate after a few hundred tokens.The arithmetic of what fits#
Intuition about what fits in two hundred thousand tokens is bad, in both directions. People assume a long contract is hopeless and it is not, and assume a mailbox will be fine and it is nowhere close. English prose runs at roughly three quarters of a word per token, which is enough to settle the question before you write any code.
def tokens_from_words(words):
"""English prose runs about 0.75 words per token. Count properly before you
ship; this is accurate enough to decide whether something is even close."""
return round(words / 0.75)
LIMIT = 200_000
SYSTEM, TOOLS, REPLY_ROOM = 1_800, 3_400, 4_000
items = [
("a 2-page memo", 2 * 500),
("a 40-page vendor contract", 40 * 450),
("a 300-page annual report", 300 * 600),
("a 90-minute call transcript", 90 * 140),
("this year's support email", 4_000 * 180),
]
print(f"{'artefact':<28}{'words':>10}{'tokens':>10}{'% of a 200k window':>21}")
for name, words in items:
t = tokens_from_words(words)
print(f"{name:<28}{words:>10,}{t:>10,}{t/LIMIT:>20.1%}")
usable = LIMIT - SYSTEM - TOOLS - REPLY_ROOM
print(f"\nwindow {LIMIT:>9,}")
print(f"system prompt -{SYSTEM:>9,}")
print(f"tool definitions -{TOOLS:>9,}")
print(f"room for the reply -{REPLY_ROOM:>9,}")
print(f"left for content {usable:>9,} ({usable/LIMIT:.1%})")
contract = tokens_from_words(40 * 450)
print(f"\nThe contract at {contract:,} tokens fits {usable // contract} times over.")
report = tokens_from_words(300 * 600)
print(f"The annual report at {report:,} does not fit at all: over by {report - usable:,}.")
# artefact words tokens % of a 200k window
# a 2-page memo 1,000 1,333 0.7%
# a 40-page vendor contract 18,000 24,000 12.0%
# a 300-page annual report 180,000 240,000 120.0%
# a 90-minute call transcript 12,600 16,800 8.4%
# this year's support email 720,000 960,000 480.0%
#
# window 200,000
# system prompt - 1,800
# tool definitions - 3,400
# room for the reply - 4,000
# left for content 190,800 (95.4%)
#
# The contract at 24,000 tokens fits 7 times over.
# The annual report at 240,000 does not fit at all: over by 49,200.A conversation is quadratic#
The window fills up gradually and the bill does not. Each turn resends everything before it, so total tokens billed grows with the square of the turn count while the conversation itself grows linearly. This is the single most surprising line on a first invoice.
SYSTEM, PER_TURN = 1_800, 600 # fixed preamble; user message plus reply per turn
IN_PER_MTOK = 3.00
PREFILL_TOK_PER_S = 8_000 # measure yours; prefill time is roughly linear
print(f"{'turn':>5}{'window':>10}{'billed so far':>15}{'$ so far':>10}{'prefill':>10}")
billed = 0
for turn in range(1, 41):
window = SYSTEM + PER_TURN * turn
billed += window
if turn in (1, 5, 10, 20, 30, 40):
print(f"{turn:>5}{window:>10,}{billed:>15,}{billed/1e6*IN_PER_MTOK:>10.3f}"
f"{window/PREFILL_TOK_PER_S:>9.2f}s")
final = SYSTEM + PER_TURN * 40
print(f"\nThe conversation ends at {final:,} tokens. You were billed for {billed:,}.")
print(f"That is {billed/final:.1f}x the size of the thing itself, "
f"because every turn resends it.")
print(f"Turn 40 costs {(SYSTEM+PER_TURN*40)/(SYSTEM+PER_TURN):.1f}x turn 1 and its prefill "
f"is {(SYSTEM+PER_TURN*40)/(SYSTEM+PER_TURN):.1f}x longer, before the model writes "
f"a word.")
# turn window billed so far $ so far prefill
# 1 2,400 2,400 0.007 0.30s
# 5 4,800 18,000 0.054 0.60s
# 10 7,800 51,000 0.153 0.97s
# 20 13,800 162,000 0.486 1.73s
# 30 19,800 333,000 0.999 2.48s
# 40 25,800 564,000 1.692 3.23s
#
# The conversation ends at 25,800 tokens. You were billed for 564,000.
# That is 21.9x the size of the thing itself, because every turn resends it.
# Turn 40 costs 10.8x turn 1 and its prefill is 10.8x longer, before the model writes a word.Cost and latency are the same curve here, which is why a long chat feels sluggish as well as costing more: the prefill at turn forty is 3.2 seconds before a single token comes back. Both are fixed by the same two moves — cache the stable prefix so the re-read is nearly free, and compact the history so it stops growing — and prompt caching only works if the history is appended to rather than rewritten, which is a constraint on how you compact.
Recall is not uniform#
A large window does not mean everything in it is equally usable. Material near the beginning and the end is recalled more reliably than material in the middle. This is well enough established to design around.
The published result is a U-shaped curve: accuracy at finding a fact is highest when it sits at the start or the end of the context, sags in the middle, and the sag deepens as the context gets longer. It is a retrieval-within-the-prompt effect rather than a limit on the window, which is why it shows up most on tasks where one specific passage carries the answer and barely at all on summarisation, where everything matters a little. The practical reading is that position is a variable you control and should be setting deliberately.
- Put the most important content near the start or the end of what you send.
- Do not bury a critical instruction in the middle of a long document.
- Fewer, better documents usually beat more documents, for accuracy as well as cost.
Rank order is the worst arrangement available, because it puts your fourth-best passage exactly where recall is weakest. Reordering costs nothing.
def sandwich(passages):
"""Rank order buries the best of the middle passages exactly where recall is
weakest. This puts the strongest at the two ends and the weakest in between."""
ranked = sorted(passages, key=lambda p: -p[1])
front, back = [], []
for i, p in enumerate(ranked):
(front if i % 2 == 0 else back).append(p)
return front + back[::-1]
passages = [(f"doc{i}", round(0.91 - 0.05 * i, 2)) for i in range(10)]
print("rank order :", [p[0] for p in passages])
print(" ", [p[1] for p in passages])
ordered = sandwich(passages)
print("sandwiched :", [p[0] for p in ordered])
print(" ", [p[1] for p in ordered])
for label, order in (("rank order", passages), ("sandwiched", ordered)):
middle = [p[1] for p in order[3:7]]
print(f"\nthe four weakest-recalled positions hold, {label}: {middle}")
print(f" best passage stranded there: {max(middle)}")
# rank order : ['doc0', 'doc1', 'doc2', 'doc3', 'doc4', 'doc5', 'doc6', 'doc7', 'doc8', 'doc9']
# [0.91, 0.86, 0.81, 0.76, 0.71, 0.66, 0.61, 0.56, 0.51, 0.46]
# sandwiched : ['doc0', 'doc2', 'doc4', 'doc6', 'doc8', 'doc9', 'doc7', 'doc5', 'doc3', 'doc1']
# [0.91, 0.81, 0.71, 0.61, 0.51, 0.46, 0.56, 0.66, 0.76, 0.86]
#
# the four weakest-recalled positions hold, rank order: [0.76, 0.71, 0.66, 0.61]
# best passage stranded there: 0.76
#
# the four weakest-recalled positions hold, sandwiched: [0.61, 0.51, 0.46, 0.56]
# best passage stranded there: 0.61What to cut, in order#
- 1.Old tool results. Rarely needed later and usually the largest single block.
- 2.Middle conversation turns, summarised into a short note.
- 3.Low-scoring retrieved documents. If your retrieval returns ten and the top three answer the question, the rest are cost.
- 4.Verbose instructions that the model already follows. Long system prompts accumulate lines nobody has tested.
Never cut the task
The original request, any user corrections, and constraints already agreed must survive every trim. An agent that loses the task keeps working confidently on the wrong problem, and nothing errors.
What to skip#
Skip building a compaction strategy for a feature whose conversations are six turns long; the arithmetic above says it is not worth the code. Skip token counting on the hot path if a character estimate is within ten per cent and you have thirty per cent headroom, and spend the call on the one place it matters, which is the boundary check before sending. And skip the instinct that a bigger window removes the problem: it moves the failure from an error you would have seen to a cost and a recall curve you will not.
Where people go wrong
Checking only the input size. If input nearly fills the window, generation has almost no room and the response truncates immediately.
If this comes up in an interview
What counts against the context window?
Everything sent plus everything generated: system prompt, tool definitions, all messages, all tool results, retrieved documents and the reply itself. Forgetting to reserve room for the reply is a common cause of failures.
Is recall uniform across a long context window?
No. Content near the start and the end is recalled more reliably than content in the middle. So position matters, and filling a large window because it exists is worse than curating what goes into it.
Why does a long conversation get expensive faster than it gets long?
Because every turn resends everything before it. The conversation grows linearly and the tokens billed grow with the square of the turn count: forty turns of a conversation that ends at 25,800 tokens bills about 564,000, roughly 22 times its own final size. Prefill latency follows the same curve, so turn 40 is also about ten times slower to start than turn 1. Prompt caching and compaction fix both, and caching only works if history is appended rather than rewritten.
What gets marked up. Strong answers name the quadratic and connect it to latency as well as cost.
A request does not fit. What do you cut first?
Old tool results, then middle conversation turns summarised, then low-scoring retrieved documents, then untested instructions. Never the original task, user corrections or agreed constraints.
Build this: Write a prompt assembler that takes a token budget and drops the lowest-value content until it fits, then prove it never drops the original task.
Watch it explained
This is one section of Context windows, 8 hours across 2 sections. Read it in full to get the whole topic in order.
Stop 13 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.