Splitting documents well
By the end: You can chunk a corpus so retrieval finds passages that actually answer questions.
- Why fixed-size splitting fails
- Splitting on structure
- Size and overlap
- Enriching chunks with context
New to chunking & document processing? A retrieval system cannot work in whole documents: a two-hundred-page manual is too large to send to a model and too coarse to match against a specific question. Chunking is cutting documents into passages small enough to retrieve and large enough to still mean something on their own. The other half of the job is getting at the text at all — pulling it out of PDFs, tables, slides and web pages without discarding the structure that gave it meaning. Why it matters
Documents have to be cut into pieces small enough to retrieve and to send, and where the cuts fall decides what can be found afterwards. This section covers splitting along the structure of a document rather than by character count, choosing a size and an overlap, and putting back the surrounding context a piece loses when it is lifted out. Get this wrong and no amount of work further down the pipeline recovers it.
Fixed size is the default and it is wrong#
Splitting every 500 characters is what most tutorials show. It cuts sentences in half, separates a heading from its content, and splits a table down the middle. The resulting chunk is not a coherent passage, so it neither embeds well nor reads well when retrieved.
chunk 41: "...customers may return items within"
chunk 42: "30 days of delivery for a full refund..."
Neither answers "how long is the return window".Split on meaning#
- 1.Split on document structure first: sections, then paragraphs.
- 2.If a section is too large, split on paragraph boundaries within it.
- 3.If a paragraph is still too large, split on sentences.
- 4.Never split mid-sentence.
How big#
There is a real tension. Small chunks retrieve precisely and often lack the surrounding context needed to be useful. Large chunks carry context and dilute the embedding, so they match everything weakly.
- Roughly 200 to 500 tokens is a reasonable starting range for prose.
- Overlap consecutive chunks by a sentence or two, so an answer spanning a boundary is not lost.
- Measure rather than guess: run the same retrieval eval at two or three sizes and compare recall.
Measure it, because the answer is not a number#
The last item on that list is the whole section, so here it is done. Two corpora of twenty documents each, containing the same facts: corpus A is reference pages with headings and short titled sections, corpus B is the same content written as flowing prose with no headings at all. Eighty questions whose answers are known exactly. Four ways of cutting, each run with and without the document title and heading path prepended before indexing.
NAMES = ["Halcyon", "Kestrel", "Marlin", "Osprey", "Petrel", "Quarry", "Redwing", "Sable",
"Tanager", "Umber", "Vireo", "Warbler", "Yarrow", "Zephyr", "Alder", "Bittern",
"Corvid", "Dunlin", "Egret", "Fulmar"]
FACTS = {nm: dict(warranty=24 + i % 4 * 12, ship=2 + i % 5, watts=45 + i * 5,
ports=2 + i % 3, temp=35 + i % 9) for i, nm in enumerate(NAMES)}
def sectioned(nm, f):
return (f"# {nm} Series reference\n\n"
f"## Overview\nThe {nm} is a desktop unit for small teams. It ships in a single "
f"carton and needs no assembly beyond attaching the stand.\n\n"
f"## Power\nThe supply draws {f['watts']} watts under continuous load. Peak draw "
f"is higher for about two seconds at start-up and the circuit should be rated for it.\n\n"
f"## Connectivity\nThere are {f['ports']} rear ports. The front panel is for "
f"service access only and should not be used for permanent connections.\n\n"
f"## Environment\nOperating temperature is rated to {f['temp']} degrees. Above "
f"that the unit throttles rather than shutting down.\n\n"
f"## Warranty\nCover runs for {f['warranty']} months from the date of delivery. "
f"Cover is void if the case has been opened by anyone outside the service network.\n\n"
f"## Returns\nGoods may be sent back within {f['ship']} weeks of delivery. The "
f"original carton is required and a refund is issued once the goods arrive.\n")
def narrative(nm, f):
return (f"The {nm} Series has been sold into small offices for several years now, and most "
f"of the questions that reach us are about the practicalities of running one. It "
f"arrives as a desktop unit in one carton, and the only assembly is the stand. The "
f"supply draws {f['watts']} watts under continuous load, which is worth checking "
f"against an older ring main before ordering more than two.\n\n"
f"On the back there are {f['ports']} ports, which is enough for most desks; the "
f"front panel exists for service access. Heat has never been a common complaint. The "
f"rating is {f['temp']} degrees and above it the behaviour is to throttle rather "
f"than to shut down, so a warm cupboard produces a slow unit rather than an outage.\n\n"
f"Cover runs for {f['warranty']} months from the date of delivery, and it is void if "
f"the case has been opened by anyone outside the service network. Goods may be sent "
f"back within {f['ship']} weeks of delivery if they are simply not wanted; the "
f"original carton is required and the refund is issued once the goods arrive back "
f"with us rather than when they are posted.\n")
QUESTIONS = []
for nm in NAMES:
f = FACTS[nm]
QUESTIONS += [(f"how long is the {nm} guarantee", f"Cover runs for {f['warranty']} months"),
(f"how many weeks to send a {nm} back", f"within {f['ship']} weeks"),
(f"what is the power draw of the {nm}", f"draws {f['watts']} watts"),
(f"how hot can a {nm} get before it slows", f"{f['temp']} degrees")]import re
def fixed(text, size, overlap=0):
step = size - overlap
return [text[i:i + size] for i in range(0, len(text), step)]
def sentences(text, target=300):
out, cur = [], ""
for s in re.split(r"(?<=[.\n])\s+", text):
if cur and len(cur) + len(s) > target:
out.append(cur.strip()); cur = ""
cur += s + " "
if cur.strip():
out.append(cur.strip())
return out
def by_section(text):
return [p.strip() for p in re.split(r"\n(?=## )", text) if p.strip()]
STRATEGIES = {"fixed 300": lambda t: fixed(t, 300),
"fixed 300/60 overlap": lambda t: fixed(t, 300, 60),
"sentence, ~300": lambda t: sentences(t, 300),
"section boundary": by_section}from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import linear_kernel
def evaluate(docs, split, enrich, k=3):
texts, indexed = [], []
for nm, doc in docs.items():
title = doc.splitlines()[0].lstrip("# ") if doc.startswith("#") else f"{nm} Series"
head = ""
for c in split(doc):
if c.startswith("## "):
head = c.splitlines()[0][3:]
texts.append(c) # what the user is shown
indexed.append(f"{title} > {head}\n{c}" if enrich else c) # what is embedded
vec = TfidfVectorizer(sublinear_tf=True).fit(indexed)
M = vec.transform(indexed)
hits = 0
for q, gold in QUESTIONS:
top = linear_kernel(vec.transform([q]), M).ravel().argsort()[::-1][:k]
hits += any(gold in texts[i] for i in top)
mean = sum(len(t) for t in texts) / len(texts)
return hits / len(QUESTIONS), len(texts), mean, mean * k
for label, build in [("A pages with headings", sectioned),
("B flowing prose, no headings", narrative)]:
docs = {nm: build(nm, FACTS[nm]) for nm in NAMES}
print(label)
print(" strategy chunks mean chars recall@3 chars sent")
for sname, split in STRATEGIES.items():
for enrich in (False, True):
r, n, avg, sent = evaluate(docs, split, enrich)
name = sname + (" + heading" if enrich else "")
print(f" {name:28s} {n:5d} {avg:11.0f} {r:9.3f} {sent:11.0f}")
# A pages with headings
# strategy chunks mean chars recall@3 chars sent
# fixed 300 60 286 0.438 857
# fixed 300 + heading 60 286 0.750 857
# fixed 300/60 overlap 80 259 0.450 777
# fixed 300/60 overlap + heading 80 259 0.750 777
# sentence, ~300 80 212 0.325 636
# sentence, ~300 + heading 80 212 0.750 636
# section boundary 140 121 0.075 362
# section boundary + heading 140 121 0.500 362
# B flowing prose, no headings
# strategy chunks mean chars recall@3 chars sent
# fixed 300 80 254 0.287 762
# fixed 300 + heading 80 254 0.500 762
# fixed 300/60 overlap 100 250 0.325 751
# fixed 300/60 overlap + heading 100 250 0.537 751
# sentence, ~300 80 252 0.200 757
# sentence, ~300 + heading 80 252 0.750 757
# section boundary 20 1014 1.000 3044
# section boundary + heading 20 1014 1.000 3044Start with the reversal, because it is the answer to the question people actually ask. Section-boundary chunking is the worst strategy on corpus A, at 0.075, and the best on corpus B, at 1.000. It is the same function. On A, a section is a two-sentence fragment averaging 121 characters, small enough that the product name has usually fallen out of it. On B there are no headings, so the function returns whole documents of 1,014 characters. The strategy did not get better; the documents changed shape underneath it. There is no best chunk size in that table and there is not one in your corpus either — there is a best chunk size for documents written the way yours are written, which is why the measurement takes an afternoon and the rule of thumb takes a career to unlearn.
The second result is larger than the first and is the one to act on. Prepending the document title and heading path before embedding is worth more than every choice of splitter combined. On corpus A it takes fixed chunks from 0.438 to 0.750 and section chunks from 0.075 to 0.500. On corpus B it takes sentence chunks from 0.200 to 0.750. Nothing else in the table moves anything by that much, and it costs a string concatenation. If you do one thing from this section, do that one.
Third, read the cost column rather than the recall column alone. On corpus B whole documents score a perfect 1.000, and they send 3,044 characters to get it. Sentence chunks with a title prepended reach 0.750 on 757 characters, a quarter of the text. Whether that trade is worth taking depends on what the prompt budget is and what the model does with a long context, but it is a trade, and a table of recall without a cost column hides it. The reason big chunks win on recall is not that they are better written, it is that they contain more, and at the limit a system that returns the entire corpus has perfect recall and is useless.
Fourth, the ceiling. No configuration on corpus A beats 0.750, and that is exactly three of the four question types. The fourth asks how long the guarantee lasts and the passage says cover runs for so many months. The two share no words at all, so no arrangement of the same words retrieves it. That is not a chunking problem and no amount of tuning chunk size will touch it; it is what hybrid search, query rewriting and a reranker are for. Knowing which of your failures are boundary failures and which are vocabulary failures decides which of those you should be working on, and the only way to know is to look at the questions that missed.
What overlap actually buys#
Overlap barely registers in the recall table above, which makes it look optional. It is not measuring the thing overlap is for. The question overlap answers is how often the answer is cut in half, and that can be counted directly.
A pages with headings
fixed 300 answer cut in half by a boundary: 20 of 80
fixed 300/60 overlap answer cut in half by a boundary: 0 of 80
sentence, ~300 answer cut in half by a boundary: 0 of 80
section boundary answer cut in half by a boundary: 0 of 80
B flowing prose
every strategy answer cut in half by a boundary: 0 of 80One in four answers is destroyed by a boundary under blind fixed-size splitting on corpus A, and sixty characters of overlap removes all of them, for a 33 per cent increase in the number of chunks. That is the trade: overlap costs storage and embedding calls in direct proportion to how much you use, and buys insurance against a failure that is total when it happens — a chunk containing half a sentence does not partially answer the question, it does not answer it at all. Splitting on sentences or sections gets the same protection for nothing, which is the better answer where the documents allow it.
Tables and code#
Both break the assumptions underneath every method above, and both are common enough in a real corpus to be worth handling explicitly rather than hoping. A table means nothing without its header row; a fixed-size cut through one produces a chunk of numbers with no column names.
TABLE = """| Plan | Monthly | Seats | Support SLA |
|------|---------|-------|-------------|
| Starter | 19 | 3 | next business day |
| Team | 79 | 25 | 8 hours |
| Business | 249 | 100 | 2 hours |
| Enterprise | 900 | unlimited | 30 minutes |
"""
def fixed(t, n):
return [t[i:i + n] for i in range(0, len(t), n)]
print(fixed(TABLE, 160)[1])
# ss | 249 | 100 | 2 hours |
# | Enterprise | 900 | unlimited | 30 minutes |
# One row per chunk, with the header carried into every one.
lines = [l for l in TABLE.strip().splitlines() if not set(l) <= set("|- ")]
header = [c.strip() for c in lines[0].strip("|").split("|")]
for row in lines[1:]:
cells = [c.strip() for c in row.strip("|").split("|")]
print("; ".join(f"{h}: {v}" for h, v in zip(header, cells)))
# Plan: Starter; Monthly: 19; Seats: 3; Support SLA: next business day
# Plan: Team; Monthly: 79; Seats: 25; Support SLA: 8 hours
# Plan: Business; Monthly: 249; Seats: 100; Support SLA: 2 hours
# Plan: Enterprise; Monthly: 900; Seats: unlimited; Support SLA: 30 minutesThe row-per-chunk form has a real cost — you lose the ability to answer a question that compares rows, such as which plan is cheapest per seat — so the usual arrangement is to index the rows for retrieval and keep the whole table as the passage that gets sent. That is the same shape as retrieving small and expanding to the parent section, and the two can share an implementation.
Code has the same problem from a different direction. A function split across two chunks is unreadable in both, and an import block or a class declaration that a snippet depends on is usually hundreds of lines away. Split code on syntactic boundaries — one function or one class per chunk, using a parser for the language rather than a regular expression — and prepend the file path and the enclosing class name, which is the code equivalent of the heading path that was worth so much above. Very long functions get split on blank lines within the body as a fallback, and a comment saying which function the fragment came from is worth adding, because a retrieved fragment with no name attached cannot be placed by anyone.
Give each chunk its context#
A chunk is retrieved alone, so it has to make sense alone. Prepending the document title and heading path before embedding measurably improves retrieval, especially on structured documents where a passage is meaningless without knowing which section it came from.
def to_embed(chunk) -> str:
return f"{chunk.doc_title} > {chunk.heading_path}\n\n{chunk.text}"
# Store the raw text separately, so what you show the user
# is the passage, not the passage plus breadcrumbs.Store metadata with every chunk
Document id, title, heading path, position, source URL and last-updated date. You need these for citations, for filtering, for re-indexing on change, and for debugging why something was retrieved.
Where people go wrong
Choosing a chunk size because a tutorial used it. Corpora differ enormously, and this is one of the few RAG parameters where measuring gives a clear answer quickly.
If this comes up in an interview
Someone asks you for the best chunk size. What do you say?
That it depends on how the documents are structured and has to be measured, and that it is usually not the biggest lever available. In a comparison across four splitters on two corpora written from the same facts, section-boundary chunking was the worst strategy on documents with headings and the best on the same content written as flowing prose, because a section means something different in each. Prepending the document title and heading path before embedding moved recall further than any choice of splitter.
Then they push further
How would you run that comparison on our corpus?
Collect twenty to fifty real questions with the passage that answers each one. Chunk the corpus three or four ways, index each, and report recall at k along with the number of characters a retrieval sends, because bigger chunks buy recall by sending more text and a recall table without a cost column hides that.
What gets marked up. Marked up for refusing to give a number, naming structure as the variable, and reporting cost alongside recall.
How would you chunk documents for retrieval?
Split on structure rather than a fixed character count: sections, then paragraphs, then sentences, never mid-sentence. Around 200 to 500 tokens with a sentence or two of overlap, and prepend the document title and heading path before embedding so each chunk is self-describing.
What is the trade-off in chunk size?
Small chunks retrieve precisely but often lack the context to be useful on their own. Large chunks carry context but dilute the embedding so they match many things weakly. The right size depends on the corpus and should be measured with a retrieval eval.
Then they push further
How would you choose the size for a particular set of documents?
Try a few sizes and measure retrieval on real questions with known source passages. The best size depends on how the documents are written and what people ask, so it is found by measurement, not by a rule.
What can you do besides picking one size?
Split on structure such as headings, attach the heading path or a short document summary to each chunk before embedding, or retrieve small chunks and pass the larger surrounding section to the model.
What gets marked up. Marked up for measuring against real questions and for structure-aware chunking. A single magic number with no reasoning is marked down.
Why overlap chunks?
So an answer that straddles a boundary is not lost from both chunks. It costs a little storage and duplication and prevents a whole class of silent retrieval failure.
Build this: Chunk the same corpus three ways: fixed 500 characters, paragraph-based, and paragraph-based with heading enrichment. Run the same twenty-question retrieval eval on each and compare recall.
Watch it explained
This is one section of Chunking & document processing, 8 hours across 2 sections. Read it in full to get the whole topic in order.
Stop 23 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.