Working with inputs bigger than you want to send

Sooner or later what you want to send does not fit, or fits and costs more than it is worth. This section is about that decision. Count the real tokens before sending rather than learning the limit from an error, give each part of the prompt a budget so you know what gives way when something grows, then choose between retrieval, staged summaries and one long call, each of which is right in different circumstances. It also covers where in a long prompt to put what matters most, because recall across a window is not uniform.

Count before you send#

Guessing token counts from characters is fine for estimates and wrong for limits. Most providers offer a token counting endpoint or library. Count the real prompt before sending, so you can decide what to cut instead of discovering it from an error.

The estimate goes wrong in one direction and always on the inputs you care about. Four characters to a token holds for English prose; it does not hold for source code, which is dense in punctuation, for JSON, where every brace and quote is its own token, for languages that do not use the Latin alphabet, which can run several times worse, or for anything base64 encoded, which tokenises close to one token per two or three characters. A retrieval corpus of API documentation can be forty per cent above the estimate, and forty per cent is the difference between comfortable and rejected.

Give every part a budget#

Window 200k tokens (example)
  system prompt and tools     6k   fixed
  retrieved documents        40k   cut lowest-ranked first
  conversation history       20k   summarise oldest turns
  user message                4k   reject beyond this, with a clear message
  reserved for the reply      8k
  headroom                   the rest, because long prompts cost and slow

Budgets turn a vague worry into a rule each part must follow. When something grows, you know what gives way.

Two details make the difference between a budget that works and one that reads well. Each line needs a named policy, not just a number, because the interesting question is what happens at the limit — the user message line above rejects with a message, the documents line drops the lowest-ranked, and those are different behaviours that have to be written somewhere. And the budget should be enforced in one function that every code path goes through, because the failure people actually hit is a second, less-travelled path that assembles the prompt its own way and has no budget at all.

Three ways to handle a long document#

• Retrieval: the question needs one part of the document. Find that part and send only it. Cheapest and usually most accurate.
• Map-reduce: the question needs all of it, such as a summary or 'list every obligation'. Process each section separately, then combine the results.
• One long call: the document fits, the question needs connections across it, and the cost is acceptable. Simplest to build, most expensive per question.

The question decides, not the document size. A question with one answer in one place is a retrieval question however long the document is. A question of the form every, all or how many is a map-reduce question, because retrieval will confidently return the top five of the eleven obligations and nothing will indicate that six are missing. A question about how one part relates to another is the case for a single long call, because that is the only arrangement in which both parts are in front of the model at the same time.

SENT = []

def ask(prompt, reply_tokens=400):
    """Stand-in for one model call, so the accounting below is real.
    In your code this is client.chat([{"role": "user", "content": prompt}])."""
    SENT.append((len(prompt) // 4, reply_tokens))
    return f"[summary of {len(prompt) // 4} tokens]"

def summarise_long(sections):
    partials = [ask("Summarise, keeping every figure and name:\n" + s) for s in sections]
    return ask("Combine these section summaries into one, "
               "keeping section references:\n" + "\n\n".join(partials))

document = "word " * 240_000                      # about 300,000 tokens
sections = [document[i:i + 80_000] for i in range(0, len(document), 80_000)]
summarise_long(sections)

sent_in = sum(a for a, _ in SENT)
sent_out = sum(b for _, b in SENT)
biggest = max(a for a, _ in SENT)
one_call = len(document) // 4

print(f"{len(sections)} sections, {len(SENT)} model calls")
print(f"largest single call  {biggest:>9,} input tokens")
print(f"total input billed   {sent_in:>9,}")
print(f"total output billed  {sent_out:>9,}")
print(f"one long call        {one_call:>9,} input tokens, in a single request")
print(f"\nMap-reduce bills {sent_in/one_call:.2f}x the input of the long call and "
      f"{sent_out/400:.0f}x the output,")
print(f"but no request exceeds {biggest:,} tokens, and the {len(sections)} map calls "
      f"run in parallel.")
print("The saving is not money. It is that the job is possible at all.")

# 15 sections, 16 model calls
# largest single call     20,010 input tokens
# total input billed     300,268
# total output billed      6,400
# one long call          300,000 input tokens, in a single request
#
# Map-reduce bills 1.00x the input of the long call and 16x the output,
# but no request exceeds 20,010 tokens, and the 15 map calls run in parallel.
# The saving is not money. It is that the job is possible at all.

The thing people expect from map-reduce is a saving, and there is not one: the same text goes through the model either way, plus the partial summaries on top. What it buys is that every request fits, that the fifteen map calls run at the same time so wall-clock latency is one section rather than fifteen, and that a failure costs you one section instead of the whole job. What it costs is the information that crossed a section boundary, which is why a clause on page four modifying a clause on page ninety is the case map-reduce quietly gets wrong.

Longer context or retrieval: the arithmetic#

This is the design question that actually comes up, and it is usually answered by taste. It does not have to be. There are three options — send everything every time, send everything but cache the prefix, or retrieve a slice — and the inputs that decide between them are the size of the corpus, how many questions you will ask it, and how long it takes to build the retrieval pipeline.

IN, OUT = 3.00, 15.00          # dollars per million tokens. Substitute yours.
EMBED_PER_MTOK = 0.02
CACHE_WRITE, CACHE_READ = 1.25, 0.10    # multipliers on the input rate
DOC, RETRIEVED = 180_000, 6_000
PROMPT, REPLY = 400, 500
BUILD_HOURS, HOURLY = 24, 90   # what the retrieval pipeline costs to build and wire up

def q_cost(context, mult=1.0):
    return (context * mult + PROMPT) / 1e6 * IN + REPLY / 1e6 * OUT

long_q = q_cost(DOC)
short_q = q_cost(RETRIEVED)
cached_q = q_cost(DOC, CACHE_READ)
index_once = DOC / 1e6 * EMBED_PER_MTOK
build_once = BUILD_HOURS * HOURLY

print(f"whole document every time      " "$" f"{long_q:>8.4f} a question")
print(f"whole document, cached prefix  " "$" f"{cached_q:>8.4f} a question "
      "(plus " "$" f"{q_cost(DOC, CACHE_WRITE) - long_q:.4f} on the first, "
      "and only while the entry lives)")
print(f"retrieve 6k of it              " "$" f"{short_q:>8.4f} a question")

print(f"\n{'questions':>10}{'long':>12}{'cached':>12}{'retrieval':>12}{'retrieval+build':>17}")
for n in (1, 100, 1_000, 10_000, 100_000):
    a, b, c = n * long_q, n * cached_q, index_once + n * short_q
    print(f"{n:>10,}{a:>12,.0f}{b:>12,.0f}{c:>12,.0f}{c + build_once:>17,.0f}")

n_tokens = build_once / (long_q - short_q)
n_cached = build_once / (cached_q - short_q)
print(f"\nOn tokens alone retrieval wins from the first question. Including the "
      f"{BUILD_HOURS} hours to build it at " "$" f"{HOURLY}/h, it wins from question "
      f"{n_tokens:,.0f}.")
print(f"Against a cached prefix it needs {n_cached:,.0f} questions, which most internal "
      f"tools never reach.")

# whole document every time      $  0.5487 a question
# whole document, cached prefix  $  0.0627 a question (plus $0.1350 on the first, and only while the entry lives)
# retrieve 6k of it              $  0.0267 a question
#
#  questions        long      cached   retrieval  retrieval+build
#          1           1           0           0            2,160
#        100          55           6           3            2,163
#      1,000         549          63          27            2,187
#     10,000       5,487         627         267            2,427
#    100,000      54,870       6,270       2,670            4,830
#
# On tokens alone retrieval wins from the first question. Including the 24 hours to build it at $90/h, it wins from question 4,138.
# Against a cached prefix it needs 60,000 questions, which most internal tools never reach.

Read the table as a decision rather than a cost comparison, because the token numbers alone say build retrieval and the token numbers alone are misleading. A corpus small enough to cache and fewer than a few thousand questions: cache the prefix and write no retrieval code, and note that this is the option nobody proposes in a design review. Many questions, or a corpus too large to sit in one window, or one that changes constantly so no cache entry survives: build retrieval, and the four thousand questions it takes to pay for itself will arrive in a week. Sending the whole document uncached every time is the one arrangement that is never right at volume, and it is the one that gets shipped, because it is what the prototype did.

Two things the arithmetic does not capture and you should say out loud in an interview. Cache entries expire in minutes, so the cached column only applies to bursts of questions, not to twenty a day spread over office hours. And retrieval that returns the wrong passage is wrong for free, whereas a long context that contains the answer is at worst expensive — which is why the accuracy comparison, not the cost one, is what an eval has to settle.

Position matters#

Models tend to use information at the start and end of a long context more reliably than information buried in the middle. Put instructions and the question where they are hard to miss, and put the most relevant retrieved passages first or last rather than in the middle of a long list.

Summaries that keep what matters#

When a conversation is compressed, a generic summary loses the details later turns depend on: ids, numbers, constraints the user stated, things already tried. Ask for those explicitly, or better, keep them in a structured record next to the summary.

Summary of turns 1 to 30:
  Goal: move Jack's team from the monthly to the annual plan
  Facts: account 4471, 12 seats, renewal date 1 March
  Decided: keep current seat count
  Tried and failed: coupon SPRING (expired)
  Open: whether annual includes the support add-on

Tried and failed is the line that earns its place. Without it a compacted agent retries the thing that did not work, sometimes repeatedly, and the loop looks like a reasoning failure when it is a memory one. Keep that record as structured fields your code writes and reads, not as prose the model regenerates each time it compacts, because a summary of a summary loses a little more on every pass and nothing tells you which pass dropped the account number.

What to skip#

Skip retrieval for a corpus that fits in the window and gets asked a few dozen questions a day; cache it instead and spend the fortnight elsewhere. Skip map-reduce when the answer lives in one place, since it pays for the whole document to answer a question about one paragraph. And skip tuning chunk sizes before you have measured whether retrieval is even the failing stage — that is the debugging bisect from the debugging topic, applied here.