Context limits and what fits
By the end: You can work out whether something will fit, and decide what to do when it does not.
- What the context window is
- That input and output share it
- What happens when you exceed it
- The first things to try when something does not fit
New to tokenization? A model does not read letters, and it does not quite read words either. Before any text reaches it, the text is cut into tokens: short pieces of characters drawn from a fixed vocabulary the model was trained on. A common word like "the" is one token; "unhappiness" might arrive as three; a space or a bracket is often a token of its own. Why it matters
There is a ceiling on how much a model can consider at once, and it is not only your question that counts against it. This section is about working out in advance whether a request will fit, and what to do first when it will not. Exceeding the ceiling is not a graceful degradation: the call fails outright, and it tends to fail on your largest and most important request rather than on a test.
The window#
A model can only consider a fixed number of tokens at once. That budget is the context window, and it covers everything: the system prompt, the conversation history, retrieved documents, tool definitions, tool results, and the answer the model is about to write.
Context window
|<------------------------------------------->|
[ system ][ history ][ documents ][ reply room ]Exceeding it does not give you a partial answer. The request is rejected. So anything that assembles a prompt from variable-length parts needs to check before sending.
Bigger is not the same as better#
Context windows have grown very large, which tempts people to stop thinking about what they send. Two reasons not to. Cost scales with everything you include, every single call. And recall is not uniform across a long window: material in the middle is retrieved less reliably than material near the beginning or the end.
The most common wrong answer in a design round
'The window is a million tokens, so just put the whole database in context.' It is expensive, it is slower, and the model attends to the middle less well. Curating what goes in beats dumping everything in.
When it does not fit#
- 1.Send less. Retrieve the relevant sections instead of the whole document. This is usually both cheaper and more accurate.
- 2.Summarise older conversation turns and keep the summary instead of the transcript.
- 3.Split the work into several calls and combine the results.
- 4.Only then reach for a model with a larger window.
The order matters. Reaching for a bigger window first solves the error and leaves you with a system that is expensive and gets worse as it grows.
Where people go wrong
Forgetting to leave room for the reply. If input fills the window, generation has nowhere to go and the call fails or truncates immediately.
If this comes up in an interview
What is a context window?
The maximum number of tokens a model can process in one request. It includes the system prompt, the full conversation history, any retrieved content, tool definitions and the response being generated.
What happens if you exceed it?
The request is rejected rather than truncated. Anything building a prompt from variable-length pieces has to check the size before sending.
If a model has a very large context window, why not put everything in it?
Cost scales with every token on every call, latency grows, and recall across a long window is uneven, with material in the middle attended to least reliably. Retrieving the relevant part is usually both cheaper and more accurate.
A user uploads a 500-page PDF and asks a question about it. How do you handle it?
Do not send all of it. Chunk the document, index it, retrieve the sections relevant to the question, and send only those with citations back to the source. That keeps cost flat as documents grow and lets you show where the answer came from.
Then they push further
The question is 'summarise the whole thing'. Does retrieval still work?
No, because the answer depends on all of it. Summarise in stages instead: summarise sections, then summarise the summaries, keeping track of which section each point came from.
What gets marked up. Strong answers separate questions that need one part of the document from ones that need all of it.
Build this: Write a function that assembles a prompt from a system message, a history and a list of documents, and drops the least important documents until the whole thing fits inside a token budget you pass in.
Watch it explained
This is one section of Tokenization, 10 hours across 4 sections. Read it in full to get the whole topic in order.
Stop 3 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.