Chunking & document processing
The unglamorous half of retrieval.
A retrieval system cannot work in whole documents: a two-hundred-page manual is too large to send to a model and too coarse to match against a specific question. Chunking is cutting documents into passages small enough to retrieve and large enough to still mean something on their own. The other half of the job is getting at the text at all — pulling it out of PDFs, tables, slides and web pages without discarding the structure that gave it meaning.
Split in the wrong place and you produce passages that cannot answer anything: a table severed from its column headings, a paragraph whose subject was named two pages earlier, a sentence cut in half. Nothing downstream recovers information destroyed here — no embedding model and no reranker can restore a heading that was thrown away.
It is the first step of the retrieval pipeline, before embeddings, the vector database and RAG, and it is the step most often skipped over. Chunk size is also a context window decision, since the passages are what you will be spending tokens on. When RAG answers are bad, this is the first place to look, not the model.
Most bad RAG answers were decided at parsing time, before any model was involved.
In more detailReal corpora are PDFs with tables, HTML with navigation cruft, and documents whose meaning depends on their headings. Splitting on a fixed token count destroys that structure; splitting on document structure preserves it. Most RAG quality problems are traceable to this stage rather than to the model.
Learn first
Getting clean text out of real documents
By the end: You can turn messy source files into text worth indexing.
- Why real documents are hard
- PDFs, HTML and tables
- What to strip and what to keep
- Deciding when a document is not worth indexing
Every retrieval tutorial starts from clean paragraphs of prose. Your corpus will be scanned PDFs, exported spreadsheets, pages wrapped in navigation, and documents whose meaning sits in a heading two pages earlier. This section is about getting usable text out of all that, and about the judgement call of which documents are not worth indexing at all. It is the part nobody demonstrates and the part where the most quality is lost.
The unglamorous half#
Retrieval demos use clean paragraphs of prose. Real corpora are scanned PDFs, HTML with three navigation menus, spreadsheets exported as text, and documents whose meaning depends entirely on their headings. Most RAG quality is lost here, before any model is involved.
PDFs#
A PDF describes where to draw glyphs, not what the text says. Extraction has to reconstruct reading order, and it frequently gets it wrong on multi-column layouts, headers, footnotes and tables.
- Check extraction output by eye on a sample before indexing thousands of documents.
- Multi-column layouts commonly interleave into nonsense. Test one.
- Tables usually collapse into a run of numbers with no structure. Either extract them separately or send the page as an image to a multimodal model.
- Scanned documents have no text layer at all and need OCR, which introduces its own errors.
The first item on that list is the one to argue with, because reading by eye does not scale past a few dozen documents and every team eventually wants an automatic check instead. It is worth knowing in advance what such a check can and cannot see. Below are three versions of the same passage: clean, a two-column page read straight across so the columns interleave, and a poor scan. Three cheap statistics are computed for each.
import random, re
rng = random.Random(0)
CLEAN = ("Cover runs for twenty four months from the date of delivery. Cover is void if the "
"case has been opened by anyone outside the service network. Goods may be sent back "
"within three weeks of delivery, and the original carton is required.")
OTHER = ("The supply draws one hundred and twenty watts under continuous load. Peak draw is "
"higher for about two seconds at start up. There are four rear ports and the front "
"panel is for service access only.")
# A two-column page read straight across: sentences from both columns interleaved.
INTERLEAVED = " ".join(a + ". " + b + ". " for a, b in zip(CLEAN.split(". "), OTHER.split(". ")))
# A poor scan: the usual character confusions, plus dropped characters.
def ocr(t):
t = t.translate(str.maketrans({"o": "0", "l": "1", "i": "1", "s": "5"}))
return "".join(c for c in t if rng.random() > 0.08)
GARBLED = ocr(CLEAN)
def stats(t):
toks = re.findall(r"[A-Za-z0-9]+", t)
return (sum(c.isalpha() for c in t) / len(t) * 100,
sum(c.isdigit() for c in t) / len(t) * 100,
sum(len(w) for w in toks) / len(toks))
print("document alpha% digit% mean word length")
for name, t in [("clean", CLEAN), ("two columns interleaved", INTERLEAVED), ("bad OCR", GARBLED)]:
a, d, m = stats(t)
print(f"{name:24s} {a:6.1f} {d:7.1f} {m:17.1f}")
# document alpha% digit% mean word length
# clean 80.7 0.0 4.5
# two columns interleaved 79.6 0.0 4.4
# bad OCR 63.0 18.7 4.7The OCR failure is obvious and cheap to catch: alphabetic characters drop from 81 per cent to 63 and digits jump from nothing to 19, because every o has become a 0 and every s a 5. A threshold on either statistic flags it, and running that threshold over a corpus is minutes of work. The interleaved page is the problem. It scores 79.6 and 0.0 and a mean word length of 4.4, which is indistinguishable from clean text, because every word in it is a real word — they are simply in an order that means something different from what the page said.
bad OCR, first 88 characters:
C0ver run5 f0r twenty f0ur m0nth5 f0m te date 0f de11very. C0ver 15 v01d 1f the ca5e ha5
two columns interleaved, first 118 characters:
Cover runs for twenty four months from the date of delivery. The supply draws one hundred
and twenty watts under contiThat is the honest state of automatic extraction checks. Character statistics catch the failures that produce nonsense, and they are worth having because those failures are common and cheap to find. Nothing cheap catches the failures that produce plausible text: reordered columns, a footnote spliced into the middle of a sentence, a table flattened into a run of numbers, a header repeated inside every page's body. For those you need either a human reading a sample, or a check that compares extracted text against something else — the page count, the position of known headings, a second extractor disagreeing with the first. Reading twenty documents by eye before indexing two thousand is not a shortcut being skipped, it is the only method that sees this class of damage.
HTML#
Strip navigation, footers, cookie banners and sidebars, or every chunk in your index contains the same menu text and retrieval scores become meaningless. Keep the heading structure, because it is the most valuable signal in the document.
The cost of not doing it is easy to put a number on. Twenty short reference pages, chunked at 400 characters, with sixty questions whose answers are known — first with the navigation stripped, then with the same navigation block pasted at the top and bottom of every page as a real export would leave it.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import linear_kernel
NAMES = ["Halcyon", "Kestrel", "Marlin", "Osprey", "Petrel", "Quarry", "Redwing", "Sable",
"Tanager", "Umber", "Vireo", "Warbler", "Yarrow", "Zephyr", "Alder", "Bittern",
"Corvid", "Dunlin", "Egret", "Fulmar"]
NAV = ("Home Products Solutions Pricing Documentation Support Company Careers Blog Contact "
"Sign in Start free trial Accept all cookies Manage preferences Privacy policy Terms "
"of service Copyright all rights reserved. ")
def page(nm, i):
return (f"# {nm} Series reference\n\n"
f"## Power\nThe supply draws {45 + i * 5} watts under continuous load. Peak draw "
f"is higher for about two seconds at start-up.\n\n"
f"## Environment\nOperating temperature is rated to {35 + i % 9} degrees. Above "
f"that the unit throttles rather than shutting down.\n\n"
f"## Warranty\nCover runs for {24 + i % 4 * 12} months from the date of delivery. "
f"Cover is void if the case has been opened outside the service network.\n")
QUESTIONS = []
for i, nm in enumerate(NAMES):
QUESTIONS += [(f"what is the power draw of the {nm}", f"draws {45 + i * 5} watts"),
(f"how hot can a {nm} get before it slows", f"{35 + i % 9} degrees"),
(f"how long is the {nm} cover", f"Cover runs for {24 + i % 4 * 12} months")]
fixed = lambda t, n: [t[i:i + n] for i in range(0, len(t), n)]
navwords = set(re.findall(r"[a-z]+", NAV.lower()))
for label, wrap in [("navigation stripped", False), ("navigation kept", True)]:
chunks = []
for i, nm in enumerate(NAMES):
doc = page(nm, i)
chunks += fixed(NAV + doc + NAV if wrap else doc, 400)
vec = TfidfVectorizer(sublinear_tf=True).fit(chunks)
M = vec.transform(chunks)
hits = sum(any(gold in chunks[j] for j in
linear_kernel(vec.transform([q]), M).ravel().argsort()[::-1][:3])
for q, gold in QUESTIONS)
pure = sum(len(set(re.findall(r"[a-z]+", c.lower())) - navwords) < 3 for c in chunks)
total = sum(len(c) for c in chunks)
menu = len(NAV) * 2 * len(NAMES) if wrap else 0
print(f"{label:21s} chunks {len(chunks):3d} recall@3 {hits/len(QUESTIONS):.3f}"
f" all-menu chunks {pure:2d} index {total:6d} chars, {menu/total*100:4.1f}% menu")
# navigation stripped chunks 20 recall@3 1.000 all-menu chunks 0 index 7910 chars, 0.0% menu
# navigation kept chunks 60 recall@3 0.500 all-menu chunks 20 index 16270 chars, 51.4% menuRecall halves, and two of the three numbers alongside it explain why. Twenty of the sixty chunks contain nothing but menu items, so a third of the index is made of passages that can never answer anything and can still be returned. And 51 per cent of the characters being embedded, stored and paid for are a navigation bar — the same navigation bar, twenty times. On a real corpus that is a meaningful share of both the bill and the latency, spent on text nobody will ever want.
The other reason to strip before chunking rather than after is that boilerplate moves the boundaries. With the navigation kept, the same twenty pages produce sixty chunks rather than twenty, and the cuts fall in different places, so a passage that was intact becomes split. Extraction and chunking are not independent steps; what you leave in the text decides where the knife lands.
Keep the structure#
Headings tell you what a passage is about. Discarding them and keeping only the prose throws away the cheapest context you will ever get.
Raw text of a chunk:
"Processed within 5 working days of receipt."
With its heading path:
"Returns policy > International orders > Refund timing
Processed within 5 working days of receipt."
The second is retrievable. The first is not.Know when to skip#
Bad extraction is worse than no document
A document that extracts into garbled text still gets embedded, still gets retrieved, and produces confident nonsense. Better to detect the failure and exclude it. A simple check on the ratio of recognisable words to total tokens catches most of these.
Where people go wrong
Indexing everything without checking extraction quality. Garbled documents embed and retrieve like any other, so the failure surfaces much later as confident nonsense with a citation attached.
Build this: Take ten real PDFs including one multi-column and one with a table. Extract them, read the output by eye, and write a check that flags the ones that came out badly.
Open this section on its own pageSplitting documents well
By the end: You can chunk a corpus so retrieval finds passages that actually answer questions.
- Why fixed-size splitting fails
- Splitting on structure
- Size and overlap
- Enriching chunks with context
Documents have to be cut into pieces small enough to retrieve and to send, and where the cuts fall decides what can be found afterwards. This section covers splitting along the structure of a document rather than by character count, choosing a size and an overlap, and putting back the surrounding context a piece loses when it is lifted out. Get this wrong and no amount of work further down the pipeline recovers it.
Fixed size is the default and it is wrong#
Splitting every 500 characters is what most tutorials show. It cuts sentences in half, separates a heading from its content, and splits a table down the middle. The resulting chunk is not a coherent passage, so it neither embeds well nor reads well when retrieved.
chunk 41: "...customers may return items within"
chunk 42: "30 days of delivery for a full refund..."
Neither answers "how long is the return window".Split on meaning#
- 1.Split on document structure first: sections, then paragraphs.
- 2.If a section is too large, split on paragraph boundaries within it.
- 3.If a paragraph is still too large, split on sentences.
- 4.Never split mid-sentence.
How big#
There is a real tension. Small chunks retrieve precisely and often lack the surrounding context needed to be useful. Large chunks carry context and dilute the embedding, so they match everything weakly.
- Roughly 200 to 500 tokens is a reasonable starting range for prose.
- Overlap consecutive chunks by a sentence or two, so an answer spanning a boundary is not lost.
- Measure rather than guess: run the same retrieval eval at two or three sizes and compare recall.
Measure it, because the answer is not a number#
The last item on that list is the whole section, so here it is done. Two corpora of twenty documents each, containing the same facts: corpus A is reference pages with headings and short titled sections, corpus B is the same content written as flowing prose with no headings at all. Eighty questions whose answers are known exactly. Four ways of cutting, each run with and without the document title and heading path prepended before indexing.
NAMES = ["Halcyon", "Kestrel", "Marlin", "Osprey", "Petrel", "Quarry", "Redwing", "Sable",
"Tanager", "Umber", "Vireo", "Warbler", "Yarrow", "Zephyr", "Alder", "Bittern",
"Corvid", "Dunlin", "Egret", "Fulmar"]
FACTS = {nm: dict(warranty=24 + i % 4 * 12, ship=2 + i % 5, watts=45 + i * 5,
ports=2 + i % 3, temp=35 + i % 9) for i, nm in enumerate(NAMES)}
def sectioned(nm, f):
return (f"# {nm} Series reference\n\n"
f"## Overview\nThe {nm} is a desktop unit for small teams. It ships in a single "
f"carton and needs no assembly beyond attaching the stand.\n\n"
f"## Power\nThe supply draws {f['watts']} watts under continuous load. Peak draw "
f"is higher for about two seconds at start-up and the circuit should be rated for it.\n\n"
f"## Connectivity\nThere are {f['ports']} rear ports. The front panel is for "
f"service access only and should not be used for permanent connections.\n\n"
f"## Environment\nOperating temperature is rated to {f['temp']} degrees. Above "
f"that the unit throttles rather than shutting down.\n\n"
f"## Warranty\nCover runs for {f['warranty']} months from the date of delivery. "
f"Cover is void if the case has been opened by anyone outside the service network.\n\n"
f"## Returns\nGoods may be sent back within {f['ship']} weeks of delivery. The "
f"original carton is required and a refund is issued once the goods arrive.\n")
def narrative(nm, f):
return (f"The {nm} Series has been sold into small offices for several years now, and most "
f"of the questions that reach us are about the practicalities of running one. It "
f"arrives as a desktop unit in one carton, and the only assembly is the stand. The "
f"supply draws {f['watts']} watts under continuous load, which is worth checking "
f"against an older ring main before ordering more than two.\n\n"
f"On the back there are {f['ports']} ports, which is enough for most desks; the "
f"front panel exists for service access. Heat has never been a common complaint. The "
f"rating is {f['temp']} degrees and above it the behaviour is to throttle rather "
f"than to shut down, so a warm cupboard produces a slow unit rather than an outage.\n\n"
f"Cover runs for {f['warranty']} months from the date of delivery, and it is void if "
f"the case has been opened by anyone outside the service network. Goods may be sent "
f"back within {f['ship']} weeks of delivery if they are simply not wanted; the "
f"original carton is required and the refund is issued once the goods arrive back "
f"with us rather than when they are posted.\n")
QUESTIONS = []
for nm in NAMES:
f = FACTS[nm]
QUESTIONS += [(f"how long is the {nm} guarantee", f"Cover runs for {f['warranty']} months"),
(f"how many weeks to send a {nm} back", f"within {f['ship']} weeks"),
(f"what is the power draw of the {nm}", f"draws {f['watts']} watts"),
(f"how hot can a {nm} get before it slows", f"{f['temp']} degrees")]import re
def fixed(text, size, overlap=0):
step = size - overlap
return [text[i:i + size] for i in range(0, len(text), step)]
def sentences(text, target=300):
out, cur = [], ""
for s in re.split(r"(?<=[.\n])\s+", text):
if cur and len(cur) + len(s) > target:
out.append(cur.strip()); cur = ""
cur += s + " "
if cur.strip():
out.append(cur.strip())
return out
def by_section(text):
return [p.strip() for p in re.split(r"\n(?=## )", text) if p.strip()]
STRATEGIES = {"fixed 300": lambda t: fixed(t, 300),
"fixed 300/60 overlap": lambda t: fixed(t, 300, 60),
"sentence, ~300": lambda t: sentences(t, 300),
"section boundary": by_section}from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import linear_kernel
def evaluate(docs, split, enrich, k=3):
texts, indexed = [], []
for nm, doc in docs.items():
title = doc.splitlines()[0].lstrip("# ") if doc.startswith("#") else f"{nm} Series"
head = ""
for c in split(doc):
if c.startswith("## "):
head = c.splitlines()[0][3:]
texts.append(c) # what the user is shown
indexed.append(f"{title} > {head}\n{c}" if enrich else c) # what is embedded
vec = TfidfVectorizer(sublinear_tf=True).fit(indexed)
M = vec.transform(indexed)
hits = 0
for q, gold in QUESTIONS:
top = linear_kernel(vec.transform([q]), M).ravel().argsort()[::-1][:k]
hits += any(gold in texts[i] for i in top)
mean = sum(len(t) for t in texts) / len(texts)
return hits / len(QUESTIONS), len(texts), mean, mean * k
for label, build in [("A pages with headings", sectioned),
("B flowing prose, no headings", narrative)]:
docs = {nm: build(nm, FACTS[nm]) for nm in NAMES}
print(label)
print(" strategy chunks mean chars recall@3 chars sent")
for sname, split in STRATEGIES.items():
for enrich in (False, True):
r, n, avg, sent = evaluate(docs, split, enrich)
name = sname + (" + heading" if enrich else "")
print(f" {name:28s} {n:5d} {avg:11.0f} {r:9.3f} {sent:11.0f}")
# A pages with headings
# strategy chunks mean chars recall@3 chars sent
# fixed 300 60 286 0.438 857
# fixed 300 + heading 60 286 0.750 857
# fixed 300/60 overlap 80 259 0.450 777
# fixed 300/60 overlap + heading 80 259 0.750 777
# sentence, ~300 80 212 0.325 636
# sentence, ~300 + heading 80 212 0.750 636
# section boundary 140 121 0.075 362
# section boundary + heading 140 121 0.500 362
# B flowing prose, no headings
# strategy chunks mean chars recall@3 chars sent
# fixed 300 80 254 0.287 762
# fixed 300 + heading 80 254 0.500 762
# fixed 300/60 overlap 100 250 0.325 751
# fixed 300/60 overlap + heading 100 250 0.537 751
# sentence, ~300 80 252 0.200 757
# sentence, ~300 + heading 80 252 0.750 757
# section boundary 20 1014 1.000 3044
# section boundary + heading 20 1014 1.000 3044Start with the reversal, because it is the answer to the question people actually ask. Section-boundary chunking is the worst strategy on corpus A, at 0.075, and the best on corpus B, at 1.000. It is the same function. On A, a section is a two-sentence fragment averaging 121 characters, small enough that the product name has usually fallen out of it. On B there are no headings, so the function returns whole documents of 1,014 characters. The strategy did not get better; the documents changed shape underneath it. There is no best chunk size in that table and there is not one in your corpus either — there is a best chunk size for documents written the way yours are written, which is why the measurement takes an afternoon and the rule of thumb takes a career to unlearn.
The second result is larger than the first and is the one to act on. Prepending the document title and heading path before embedding is worth more than every choice of splitter combined. On corpus A it takes fixed chunks from 0.438 to 0.750 and section chunks from 0.075 to 0.500. On corpus B it takes sentence chunks from 0.200 to 0.750. Nothing else in the table moves anything by that much, and it costs a string concatenation. If you do one thing from this section, do that one.
Third, read the cost column rather than the recall column alone. On corpus B whole documents score a perfect 1.000, and they send 3,044 characters to get it. Sentence chunks with a title prepended reach 0.750 on 757 characters, a quarter of the text. Whether that trade is worth taking depends on what the prompt budget is and what the model does with a long context, but it is a trade, and a table of recall without a cost column hides it. The reason big chunks win on recall is not that they are better written, it is that they contain more, and at the limit a system that returns the entire corpus has perfect recall and is useless.
Fourth, the ceiling. No configuration on corpus A beats 0.750, and that is exactly three of the four question types. The fourth asks how long the guarantee lasts and the passage says cover runs for so many months. The two share no words at all, so no arrangement of the same words retrieves it. That is not a chunking problem and no amount of tuning chunk size will touch it; it is what hybrid search, query rewriting and a reranker are for. Knowing which of your failures are boundary failures and which are vocabulary failures decides which of those you should be working on, and the only way to know is to look at the questions that missed.
What overlap actually buys#
Overlap barely registers in the recall table above, which makes it look optional. It is not measuring the thing overlap is for. The question overlap answers is how often the answer is cut in half, and that can be counted directly.
A pages with headings
fixed 300 answer cut in half by a boundary: 20 of 80
fixed 300/60 overlap answer cut in half by a boundary: 0 of 80
sentence, ~300 answer cut in half by a boundary: 0 of 80
section boundary answer cut in half by a boundary: 0 of 80
B flowing prose
every strategy answer cut in half by a boundary: 0 of 80One in four answers is destroyed by a boundary under blind fixed-size splitting on corpus A, and sixty characters of overlap removes all of them, for a 33 per cent increase in the number of chunks. That is the trade: overlap costs storage and embedding calls in direct proportion to how much you use, and buys insurance against a failure that is total when it happens — a chunk containing half a sentence does not partially answer the question, it does not answer it at all. Splitting on sentences or sections gets the same protection for nothing, which is the better answer where the documents allow it.
Tables and code#
Both break the assumptions underneath every method above, and both are common enough in a real corpus to be worth handling explicitly rather than hoping. A table means nothing without its header row; a fixed-size cut through one produces a chunk of numbers with no column names.
TABLE = """| Plan | Monthly | Seats | Support SLA |
|------|---------|-------|-------------|
| Starter | 19 | 3 | next business day |
| Team | 79 | 25 | 8 hours |
| Business | 249 | 100 | 2 hours |
| Enterprise | 900 | unlimited | 30 minutes |
"""
def fixed(t, n):
return [t[i:i + n] for i in range(0, len(t), n)]
print(fixed(TABLE, 160)[1])
# ss | 249 | 100 | 2 hours |
# | Enterprise | 900 | unlimited | 30 minutes |
# One row per chunk, with the header carried into every one.
lines = [l for l in TABLE.strip().splitlines() if not set(l) <= set("|- ")]
header = [c.strip() for c in lines[0].strip("|").split("|")]
for row in lines[1:]:
cells = [c.strip() for c in row.strip("|").split("|")]
print("; ".join(f"{h}: {v}" for h, v in zip(header, cells)))
# Plan: Starter; Monthly: 19; Seats: 3; Support SLA: next business day
# Plan: Team; Monthly: 79; Seats: 25; Support SLA: 8 hours
# Plan: Business; Monthly: 249; Seats: 100; Support SLA: 2 hours
# Plan: Enterprise; Monthly: 900; Seats: unlimited; Support SLA: 30 minutesThe row-per-chunk form has a real cost — you lose the ability to answer a question that compares rows, such as which plan is cheapest per seat — so the usual arrangement is to index the rows for retrieval and keep the whole table as the passage that gets sent. That is the same shape as retrieving small and expanding to the parent section, and the two can share an implementation.
Code has the same problem from a different direction. A function split across two chunks is unreadable in both, and an import block or a class declaration that a snippet depends on is usually hundreds of lines away. Split code on syntactic boundaries — one function or one class per chunk, using a parser for the language rather than a regular expression — and prepend the file path and the enclosing class name, which is the code equivalent of the heading path that was worth so much above. Very long functions get split on blank lines within the body as a fallback, and a comment saying which function the fragment came from is worth adding, because a retrieved fragment with no name attached cannot be placed by anyone.
Give each chunk its context#
A chunk is retrieved alone, so it has to make sense alone. Prepending the document title and heading path before embedding measurably improves retrieval, especially on structured documents where a passage is meaningless without knowing which section it came from.
def to_embed(chunk) -> str:
return f"{chunk.doc_title} > {chunk.heading_path}\n\n{chunk.text}"
# Store the raw text separately, so what you show the user
# is the passage, not the passage plus breadcrumbs.Store metadata with every chunk
Document id, title, heading path, position, source URL and last-updated date. You need these for citations, for filtering, for re-indexing on change, and for debugging why something was retrieved.
Where people go wrong
Choosing a chunk size because a tutorial used it. Corpora differ enormously, and this is one of the few RAG parameters where measuring gives a clear answer quickly.
Build this: Chunk the same corpus three ways: fixed 500 characters, paragraph-based, and paragraph-based with heading enrichment. Run the same twenty-question retrieval eval on each and compare recall.
Open this section on its own pageWhere you use this
A topic is easier to care about when you can see what it is for. These projects lean on it at a named stage.
- Stage 1, Chunks you can point at: Getting usable text out of PDFs and HTML before any of this is worth doing.
Now actually do it
Reading a topic is not learning it. These are the hours that count, and each one tells you when you are finished.
1Chunk three ways and measure
4hFixed size, structural, and structural with heading path enrichment before embedding. Same 20-question retrieval eval on each.
Done when: recall@5 for all three, and you can say which won and by how much.
2Catch a bad extraction
4hProcess ten real PDFs including a multi-column one and one with tables. Write a check that flags documents that extracted badly, and confirm it catches the bad ones.
Done when: Your pipeline refuses to index documents it cannot parse properly.
8 hours of practice
Words used here
1 term this topic teaches, each with a one-line definition if one of them stopped you.
- Chunking
- Cutting documents into passages small enough to embed and retrieve usefully. Chunk boundaries decide what a retriever can ever return, so bad splits cap the quality of everything downstream.
Interview questions
7 questions, grouped by the section that teaches the answer. An answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Getting clean text out of real documents
Why is document parsing the hard part of RAG?
Real documents are PDFs with multi-column layouts and tables, HTML full of navigation, and scans with no text layer. Extraction frequently produces the wrong reading order or loses structure, and everything downstream inherits that damage.
How do you handle tables in documents?
Extract them separately rather than letting them flatten into a run of numbers, or send the page to a multimodal model as an image. A flattened table is worse than useless because it still embeds and still retrieves.
What would you strip from HTML before indexing?
Navigation, footers, cookie banners and sidebars, because repeated boilerplate appears in every chunk and destroys the discriminating power of retrieval. Keep the heading structure, which is the most useful context available.
Splitting documents well
Someone asks you for the best chunk size. What do you say?
That it depends on how the documents are structured and has to be measured, and that it is usually not the biggest lever available. In a comparison across four splitters on two corpora written from the same facts, section-boundary chunking was the worst strategy on documents with headings and the best on the same content written as flowing prose, because a section means something different in each. Prepending the document title and heading path before embedding moved recall further than any choice of splitter.
Then they push further
How would you run that comparison on our corpus?
Collect twenty to fifty real questions with the passage that answers each one. Chunk the corpus three or four ways, index each, and report recall at k along with the number of characters a retrieval sends, because bigger chunks buy recall by sending more text and a recall table without a cost column hides that.
What gets marked up. Marked up for refusing to give a number, naming structure as the variable, and reporting cost alongside recall.
How would you chunk documents for retrieval?
Split on structure rather than a fixed character count: sections, then paragraphs, then sentences, never mid-sentence. Around 200 to 500 tokens with a sentence or two of overlap, and prepend the document title and heading path before embedding so each chunk is self-describing.
What is the trade-off in chunk size?
Small chunks retrieve precisely but often lack the context to be useful on their own. Large chunks carry context but dilute the embedding so they match many things weakly. The right size depends on the corpus and should be measured with a retrieval eval.
Then they push further
How would you choose the size for a particular set of documents?
Try a few sizes and measure retrieval on real questions with known source passages. The best size depends on how the documents are written and what people ask, so it is found by measurement, not by a rule.
What can you do besides picking one size?
Split on structure such as headings, attach the heading path or a short document summary to each chunk before embedding, or retrieve small chunks and pass the larger surrounding section to the model.
What gets marked up. Marked up for measuring against real questions and for structure-aware chunking. A single magic number with no reasoning is marked down.
Why overlap chunks?
So an answer that straddles a boundary is not lost from both chunks. It costs a little storage and duplication and prevents a whole class of silent retrieval failure.
Resources
Free to watch and read, picked for this topic.
For getting clean text out of real documents
For splitting documents well
Full courses: the whole topic, start to finish
Stop 23 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.