Getting clean text out of real documents
By the end: You can turn messy source files into text worth indexing.
- Why real documents are hard
- PDFs, HTML and tables
- What to strip and what to keep
- Deciding when a document is not worth indexing
New to chunking & document processing? A retrieval system cannot work in whole documents: a two-hundred-page manual is too large to send to a model and too coarse to match against a specific question. Chunking is cutting documents into passages small enough to retrieve and large enough to still mean something on their own. The other half of the job is getting at the text at all — pulling it out of PDFs, tables, slides and web pages without discarding the structure that gave it meaning. Why it matters
Every retrieval tutorial starts from clean paragraphs of prose. Your corpus will be scanned PDFs, exported spreadsheets, pages wrapped in navigation, and documents whose meaning sits in a heading two pages earlier. This section is about getting usable text out of all that, and about the judgement call of which documents are not worth indexing at all. It is the part nobody demonstrates and the part where the most quality is lost.
The unglamorous half#
Retrieval demos use clean paragraphs of prose. Real corpora are scanned PDFs, HTML with three navigation menus, spreadsheets exported as text, and documents whose meaning depends entirely on their headings. Most RAG quality is lost here, before any model is involved.
PDFs#
A PDF describes where to draw glyphs, not what the text says. Extraction has to reconstruct reading order, and it frequently gets it wrong on multi-column layouts, headers, footnotes and tables.
- Check extraction output by eye on a sample before indexing thousands of documents.
- Multi-column layouts commonly interleave into nonsense. Test one.
- Tables usually collapse into a run of numbers with no structure. Either extract them separately or send the page as an image to a multimodal model.
- Scanned documents have no text layer at all and need OCR, which introduces its own errors.
The first item on that list is the one to argue with, because reading by eye does not scale past a few dozen documents and every team eventually wants an automatic check instead. It is worth knowing in advance what such a check can and cannot see. Below are three versions of the same passage: clean, a two-column page read straight across so the columns interleave, and a poor scan. Three cheap statistics are computed for each.
import random, re
rng = random.Random(0)
CLEAN = ("Cover runs for twenty four months from the date of delivery. Cover is void if the "
"case has been opened by anyone outside the service network. Goods may be sent back "
"within three weeks of delivery, and the original carton is required.")
OTHER = ("The supply draws one hundred and twenty watts under continuous load. Peak draw is "
"higher for about two seconds at start up. There are four rear ports and the front "
"panel is for service access only.")
# A two-column page read straight across: sentences from both columns interleaved.
INTERLEAVED = " ".join(a + ". " + b + ". " for a, b in zip(CLEAN.split(". "), OTHER.split(". ")))
# A poor scan: the usual character confusions, plus dropped characters.
def ocr(t):
t = t.translate(str.maketrans({"o": "0", "l": "1", "i": "1", "s": "5"}))
return "".join(c for c in t if rng.random() > 0.08)
GARBLED = ocr(CLEAN)
def stats(t):
toks = re.findall(r"[A-Za-z0-9]+", t)
return (sum(c.isalpha() for c in t) / len(t) * 100,
sum(c.isdigit() for c in t) / len(t) * 100,
sum(len(w) for w in toks) / len(toks))
print("document alpha% digit% mean word length")
for name, t in [("clean", CLEAN), ("two columns interleaved", INTERLEAVED), ("bad OCR", GARBLED)]:
a, d, m = stats(t)
print(f"{name:24s} {a:6.1f} {d:7.1f} {m:17.1f}")
# document alpha% digit% mean word length
# clean 80.7 0.0 4.5
# two columns interleaved 79.6 0.0 4.4
# bad OCR 63.0 18.7 4.7The OCR failure is obvious and cheap to catch: alphabetic characters drop from 81 per cent to 63 and digits jump from nothing to 19, because every o has become a 0 and every s a 5. A threshold on either statistic flags it, and running that threshold over a corpus is minutes of work. The interleaved page is the problem. It scores 79.6 and 0.0 and a mean word length of 4.4, which is indistinguishable from clean text, because every word in it is a real word — they are simply in an order that means something different from what the page said.
bad OCR, first 88 characters:
C0ver run5 f0r twenty f0ur m0nth5 f0m te date 0f de11very. C0ver 15 v01d 1f the ca5e ha5
two columns interleaved, first 118 characters:
Cover runs for twenty four months from the date of delivery. The supply draws one hundred
and twenty watts under contiThat is the honest state of automatic extraction checks. Character statistics catch the failures that produce nonsense, and they are worth having because those failures are common and cheap to find. Nothing cheap catches the failures that produce plausible text: reordered columns, a footnote spliced into the middle of a sentence, a table flattened into a run of numbers, a header repeated inside every page's body. For those you need either a human reading a sample, or a check that compares extracted text against something else — the page count, the position of known headings, a second extractor disagreeing with the first. Reading twenty documents by eye before indexing two thousand is not a shortcut being skipped, it is the only method that sees this class of damage.
HTML#
Strip navigation, footers, cookie banners and sidebars, or every chunk in your index contains the same menu text and retrieval scores become meaningless. Keep the heading structure, because it is the most valuable signal in the document.
The cost of not doing it is easy to put a number on. Twenty short reference pages, chunked at 400 characters, with sixty questions whose answers are known — first with the navigation stripped, then with the same navigation block pasted at the top and bottom of every page as a real export would leave it.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import linear_kernel
NAMES = ["Halcyon", "Kestrel", "Marlin", "Osprey", "Petrel", "Quarry", "Redwing", "Sable",
"Tanager", "Umber", "Vireo", "Warbler", "Yarrow", "Zephyr", "Alder", "Bittern",
"Corvid", "Dunlin", "Egret", "Fulmar"]
NAV = ("Home Products Solutions Pricing Documentation Support Company Careers Blog Contact "
"Sign in Start free trial Accept all cookies Manage preferences Privacy policy Terms "
"of service Copyright all rights reserved. ")
def page(nm, i):
return (f"# {nm} Series reference\n\n"
f"## Power\nThe supply draws {45 + i * 5} watts under continuous load. Peak draw "
f"is higher for about two seconds at start-up.\n\n"
f"## Environment\nOperating temperature is rated to {35 + i % 9} degrees. Above "
f"that the unit throttles rather than shutting down.\n\n"
f"## Warranty\nCover runs for {24 + i % 4 * 12} months from the date of delivery. "
f"Cover is void if the case has been opened outside the service network.\n")
QUESTIONS = []
for i, nm in enumerate(NAMES):
QUESTIONS += [(f"what is the power draw of the {nm}", f"draws {45 + i * 5} watts"),
(f"how hot can a {nm} get before it slows", f"{35 + i % 9} degrees"),
(f"how long is the {nm} cover", f"Cover runs for {24 + i % 4 * 12} months")]
fixed = lambda t, n: [t[i:i + n] for i in range(0, len(t), n)]
navwords = set(re.findall(r"[a-z]+", NAV.lower()))
for label, wrap in [("navigation stripped", False), ("navigation kept", True)]:
chunks = []
for i, nm in enumerate(NAMES):
doc = page(nm, i)
chunks += fixed(NAV + doc + NAV if wrap else doc, 400)
vec = TfidfVectorizer(sublinear_tf=True).fit(chunks)
M = vec.transform(chunks)
hits = sum(any(gold in chunks[j] for j in
linear_kernel(vec.transform([q]), M).ravel().argsort()[::-1][:3])
for q, gold in QUESTIONS)
pure = sum(len(set(re.findall(r"[a-z]+", c.lower())) - navwords) < 3 for c in chunks)
total = sum(len(c) for c in chunks)
menu = len(NAV) * 2 * len(NAMES) if wrap else 0
print(f"{label:21s} chunks {len(chunks):3d} recall@3 {hits/len(QUESTIONS):.3f}"
f" all-menu chunks {pure:2d} index {total:6d} chars, {menu/total*100:4.1f}% menu")
# navigation stripped chunks 20 recall@3 1.000 all-menu chunks 0 index 7910 chars, 0.0% menu
# navigation kept chunks 60 recall@3 0.500 all-menu chunks 20 index 16270 chars, 51.4% menuRecall halves, and two of the three numbers alongside it explain why. Twenty of the sixty chunks contain nothing but menu items, so a third of the index is made of passages that can never answer anything and can still be returned. And 51 per cent of the characters being embedded, stored and paid for are a navigation bar — the same navigation bar, twenty times. On a real corpus that is a meaningful share of both the bill and the latency, spent on text nobody will ever want.
The other reason to strip before chunking rather than after is that boilerplate moves the boundaries. With the navigation kept, the same twenty pages produce sixty chunks rather than twenty, and the cuts fall in different places, so a passage that was intact becomes split. Extraction and chunking are not independent steps; what you leave in the text decides where the knife lands.
Keep the structure#
Headings tell you what a passage is about. Discarding them and keeping only the prose throws away the cheapest context you will ever get.
Raw text of a chunk:
"Processed within 5 working days of receipt."
With its heading path:
"Returns policy > International orders > Refund timing
Processed within 5 working days of receipt."
The second is retrievable. The first is not.Know when to skip#
Bad extraction is worse than no document
A document that extracts into garbled text still gets embedded, still gets retrieved, and produces confident nonsense. Better to detect the failure and exclude it. A simple check on the ratio of recognisable words to total tokens catches most of these.
Where people go wrong
Indexing everything without checking extraction quality. Garbled documents embed and retrieve like any other, so the failure surfaces much later as confident nonsense with a citation attached.
If this comes up in an interview
Why is document parsing the hard part of RAG?
Real documents are PDFs with multi-column layouts and tables, HTML full of navigation, and scans with no text layer. Extraction frequently produces the wrong reading order or loses structure, and everything downstream inherits that damage.
How do you handle tables in documents?
Extract them separately rather than letting them flatten into a run of numbers, or send the page to a multimodal model as an image. A flattened table is worse than useless because it still embeds and still retrieves.
What would you strip from HTML before indexing?
Navigation, footers, cookie banners and sidebars, because repeated boilerplate appears in every chunk and destroys the discriminating power of retrieval. Keep the heading structure, which is the most useful context available.
Build this: Take ten real PDFs including one multi-column and one with a table. Extract them, read the output by eye, and write a check that flags the ones that came out badly.
Watch it explained
This is one section of Chunking & document processing, 8 hours across 2 sections. Read it in full to get the whole topic in order.
Stop 23 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.