Data is the model
By the end: You can explain why data curation decides model quality more than architecture does.
- What goes into a pretraining corpus
- Deduplication and filtering
- Data mixture
- Contamination
New to pretraining? Pretraining is the first and by far the largest stage of making a language model. The model is shown an enormous quantity of text — a good share of the public internet, plus books and code — and set one repetitive task: predict what comes next. Doing that well billions of times forces it to absorb grammar, facts, styles of argument and the conventions of code, and that is where essentially all of a model's knowledge comes from. Why it matters
Architectures have largely converged; corpora have not. Most of what separates a strong model from a weak one is what went into it — what was collected, what was thrown out, and in what proportions. This section covers deduplication and filtering, the mixture question of how much code, how much maths and how many languages, and contamination, where benchmark data leaks into training and quietly voids your evaluation. It gets discussed less than architecture because it is less publishable and more commercially sensitive, not because it matters less.
The quiet majority of the work#
Architectures have converged; data has not. Much of what separates strong models from weak ones is the corpus: what was included, what was removed, and how it was balanced. This is less discussed than architecture because it is less publishable and more commercially sensitive.
There is a reason data wins that has nothing to do with fashion, and it is worth being precise about it. Compute for a training run is roughly six FLOPs per parameter per token, so the cost is linear in the number of tokens you train on. A filter that reaches the same loss on half the tokens has therefore halved the cost of the run — it is worth exactly as much as doubling the size of your cluster, and it costs a few hundred CPU-hours rather than a few million dollars. Architecture changes move the loss by a few per cent. Data changes move how many tokens you need, which moves the whole budget.
The other half of the argument is that data decides what the model is good at, which architecture does not. A better attention variant makes the same model slightly better at everything. Adding a hundred billion tokens of code makes it much better at code and, awkwardly for anyone hoping for a clean story, somewhat better at reasoning tasks that have nothing to do with programming.
Deduplication, and what duplicates actually cost#
The web repeats itself. Mirrored sites, syndicated articles, template pages, quotations, and the same licence text on ten million pages. Left in, a document that appears a thousand times gets a thousand times the influence on the weights, and there are three separate costs to that. Compute is spent seeing the same thing again. The model memorises rather than generalises, because reproducing an exact string it has seen a thousand times is the loss-minimising move. And the mixture you carefully chose is silently overwritten by whatever the web happens to copy most.
All three are measurable at a scale that fits on one machine. Take five hundred ordinary sentences and one boilerplate page mirrored two hundred times, fit the same counting model with and without the duplicates, and score it on the boilerplate and on a sentence it has never seen.
import math, random, collections
random.seed(1)
subj = ["the report", "the server", "the invoice", "the driver", "the parcel"]
verb = ["failed", "arrived", "expired", "returned", "stalled"]
when = ["on monday", "at noon", "last week", "overnight", "in transit"]
def sentence():
return f"{random.choice(subj)} {random.choice(verb)} {random.choice(when)} ."
unique = [sentence() for _ in range(500)]
boiler = "this page requires javascript to display correctly ."
corpus = unique + [boiler] * 200 # one boilerplate page, mirrored 200 times
def fit(docs):
c = collections.defaultdict(collections.Counter)
for d in docs:
t = ["<s>"] + d.split()
for a, b in zip(t, t[1:]):
c[a][b] += 1
return c
def loss(model, doc, vocab):
t = ["<s>"] + doc.split()
total = 0.0
for a, b in zip(t, t[1:]):
c = model[a]
total += -math.log((c[b] + 1) / (sum(c.values()) + vocab))
return total / (len(t) - 1)
vocab = len(set(" ".join(corpus).split())) + 1
with_dupes = fit(corpus)
deduped = fit(unique + [boiler])
held_out = sentence()
print(f"corpus {len(corpus)} documents, {sum(len(d.split()) for d in corpus)} tokens")
print(f"share of tokens that are the one mirrored page: "
f"{200*len(boiler.split())/sum(len(d.split()) for d in corpus):.1%}")
print(" boilerplate held-out sentence")
print(f"with duplicates {loss(with_dupes, boiler, vocab):11.3f} {loss(with_dupes, held_out, vocab):18.3f}")
print(f"deduplicated {loss(deduped, boiler, vocab):11.3f} {loss(deduped, held_out, vocab):18.3f}")
# corpus 700 documents, 4488 tokens
# share of tokens that are the one mirrored page: 35.7%
# boilerplate held-out sentence
# with duplicates 0.275 1.136
# deduplicated 3.067 1.072Finding the duplicates is where it gets awkward. Hashing catches exact copies and nothing else, and the web's duplication is mostly not exact: the same template with a different order number, the same article with a different date line. The standard answer is to compare sets of overlapping word sequences, called shingles, and treat two documents as near-duplicates when their Jaccard similarity passes a threshold, with MinHash used to make that comparison affordable across billions of documents.
import hashlib
A = ("Order 41823 shipped on 4 March. Track your parcel using the reference above. "
"Delivery is normally two working days. Contact support if it has not arrived.")
B = A.replace("41823", "41824").replace("4 March", "5 March") # same template, new order
C = A # a true mirror
D = ("The harbour was rebuilt in 1892 after a storm destroyed the old wooden pier. "
"Records show the council borrowed heavily to pay for the stone replacement.")
pages = {"A": A, "B": B, "C": C, "D": D}
seen, removed = set(), []
for name, p in pages.items():
h = hashlib.sha256(p.encode()).hexdigest()
if h in seen:
removed.append(name)
seen.add(h)
print("exact hashing removes:", removed, "- B survives, and B is a template")
def shingles(t, k):
w = t.lower().replace(".", "").split()
return {tuple(w[i:i+k]) for i in range(len(w) - k + 1)}
print()
print("Jaccard of A against B, by shingle size")
for k in [3, 5, 8, 13]:
a, b = shingles(A, k), shingles(B, k)
j = len(a & b) / len(a | b)
print(f" k={k:2} {j:.2f} {'caught at 0.8' if j >= 0.8 else 'slips through at 0.8'}")
# exact hashing removes: ['C'] - B survives, and B is a template
#
# Jaccard of A against B, by shingle size
# k= 3 0.66 slips through at 0.8
# k= 5 0.63 slips through at 0.8
# k= 8 0.58 slips through at 0.8
# k=13 0.47 slips through at 0.8That is the part worth understanding rather than memorising, because it decides whether your deduplication does anything. Near-duplicate detection is a ratio between the shared part and the varying part, so the same two-token difference is invisible in a long page and fatal in a short one.
SHARED = [
"Track your parcel using the reference above.",
"Delivery is normally two working days.",
"Contact support if the parcel has not arrived.",
"Returns are free within thirty days of delivery.",
"This message was sent automatically, please do not reply.",
"Our opening hours are nine until five on weekdays.",
"Gift receipts can be requested from the orders page.",
"Packaging is recycled wherever local facilities allow it.",
]
def shingles(t, k=5):
w = t.lower().replace(".", "").replace(",", "").split()
return {tuple(w[i:i+k]) for i in range(len(w) - k + 1)}
print("shared lines words Jaccard(A,B) caught at 0.8")
for n in [1, 2, 4, 8]:
tail = " ".join(SHARED[:n])
a = "Order 41823 shipped on 4 March. " + tail
b = "Order 41824 shipped on 5 March. " + tail
j = len(shingles(a) & shingles(b)) / len(shingles(a) | shingles(b))
print(f"{n:12} {len(a.split()):5} {j:12.2f} {'yes' if j >= 0.8 else 'no'}")
# shared lines words Jaccard(A,B) caught at 0.8
# 1 13 0.29 no
# 2 19 0.50 no
# 4 35 0.72 no
# 8 70 0.86 yesQuality filtering#
After deduplication the corpus is still mostly not writing. It is navigation bars, cookie banners, keyword spam, tables of numbers, autogenerated listings and pages that are one sentence and a link. The first pass is cheap heuristics, each one a single statistic with a threshold, and it is worth writing them out because it makes the shape of the problem obvious.
docs = {
"article": "The harbour was rebuilt in 1892 after a storm destroyed the old "
"wooden pier. Records from the period show that the town council "
"borrowed heavily to pay for the stone replacement, and the debt "
"was not cleared until after the war. The new pier was longer than "
"the one it replaced, and the fishing fleet grew by a third in the "
"decade that followed. A plaque at the landward end still lists the "
"names of the councillors who signed for the loan, and the date of "
"the storm that made it necessary.",
"navigation": "Home | Products | About | Contact | Login | Register | Terms | "
"Privacy | Cookies | Home | Products | About | Contact | Home | "
"Products | About | Contact | Login | Register | Terms | Privacy",
"keyword spam": "cheap flights cheap flights cheap flights book cheap flights "
"cheap flights cheap flights cheap flights now cheap flights "
"cheap flights book cheap flights cheap flights cheap flights "
"cheap flights cheap flights cheap flights cheap flights now",
"table": "2019 4.1 3.9 | 2020 4.4 4.0 | 2021 4.8 4.2 | 2022 5.1 4.6 | "
"2023 5.5 4.9 | 2024 5.9 5.1 | 2025 6.2 5.4 | 2026 6.6 5.8 | "
"2027 7.0 6.1 | 2028 7.4 6.4 | 2029 7.8 6.7 | 2030 8.2 7.0 | "
"2031 8.6 7.3 | 2032 9.0 7.6 | 2033 9.4 7.9 | 2034 9.8 8.2",
"truncated": "Click here to continue reading",
}
STOP = {"the", "and", "of", "to", "in", "a", "was", "after", "that", "it", "at"}
def flags(t):
words = t.split()
alpha = sum(c.isalpha() or c.isspace() for c in t) / len(t)
stop = sum(w.lower().strip(".,") in STOP for w in words) / len(words)
uniq = len(set(w.lower() for w in words)) / len(words)
return {"too short": len(words) < 50, "symbol heavy": alpha < 0.85,
"few stop words": stop < 0.10, "repetitive": uniq < 0.5}
names = list(flags(docs["article"]))
print(f"{'document':13} {'words':>5} " + " ".join(f"{k:>15}" for k in names))
for name, t in docs.items():
f = flags(t)
print(f"{name:13} {len(t.split()):5} " +
" ".join(f"{('DROP' if v else '-'):>15}" for v in f.values()))
print()
print("length threshold sweep: how many of the 5 documents survive it alone")
for k in [10, 50, 100, 200]:
kept = [n for n, t in docs.items() if len(t.split()) >= k]
print(f" >= {k:3} words keeps {len(kept)} {', '.join(kept) or 'nothing'}")
# document words too short symbol heavy few stop words repetitive
# article 92 - - - -
# navigation 41 DROP - DROP DROP
# keyword spam 36 DROP - DROP DROP
# table 63 - DROP DROP -
# truncated 5 DROP - - -
#
# length threshold sweep: how many of the 5 documents survive it alone
# >= 10 words keeps 4 article, navigation, keyword spam, table
# >= 50 words keeps 2 article, table
# >= 100 words keeps 0 nothing
# >= 200 words keeps 0 nothingThe sweep at the bottom is the part that costs people a corpus. A length threshold is the cheapest filter and the most destructive one, because it is the only filter that discards good text in proportion to how much of it is short. Move it from fifty words to a hundred here and everything is gone. On a real crawl the same move quietly removes most reference material, most question-and-answer pages and most code comments, and the loss curve will not tell you, because the run still trains perfectly well on what is left.
The second pass is a learned classifier. Label a sample of pages for whether they look like something worth learning from — public work such as FineWeb used a larger model to produce those labels — train a small, fast classifier on the labels, and score the whole corpus with it. The reported result is the one that matters for planning: keeping the pages a classifier judges educational gave a large improvement on knowledge and reasoning benchmarks at the same number of training tokens. Same compute, same architecture, better model.
Safety and legal filtering#
A third pass removes what should not be learned regardless of how well written it is: personal information such as names beside addresses, phone numbers, identity numbers and credentials; material that is illegal to hold; and content the model should not become fluent in producing. It is the least interesting stage technically and the one with the most consequences, because anything left in is memorised and can be reproduced, and removing it after training is not possible.
- Personal data: pattern matching for the structured kinds, classifiers for the rest, and accepting that recall will not be one.
- Credentials and keys: scraped code is full of them, and a model that has memorised a live API key is a liability rather than a curiosity.
- Licensing and provenance: what you are permitted to train on is a legal question with live disagreement, and the answer determines whether a model can be released at all.
- Harmful content: removed at this stage as well as being addressed in post-training, because a capability never learned is more reliably absent than one trained against afterwards.
Mixture#
How much code, how much maths, how many languages, how much conversation. The proportions shape capability directly. Adding more code improves reasoning in ways that extend beyond programming, which is one of the more surprising findings.
There is no theory for choosing them. The method everyone uses is to train several small proxy models, one per candidate mixture, evaluate them on the tasks you care about and scale up the winner. It works because the ranking of mixtures is reasonably stable across scale even though the absolute scores are not, and it is affordable because the proxy models are a thousandth of the size. If asked how you would pick a mixture, that answer — train small, compare, scale the winner — is the one being looked for, and 'principled reasoning about proportions' is not.
Contamination#
Contamination runs both ways
If benchmark questions are in the training data, benchmark scores measure memory. Serious labs try to remove evaluation data from pretraining corpora, and detecting what leaked is genuinely difficult.
The standard defence is n-gram matching: take every benchmark question, cut it into overlapping sequences of thirteen words, and drop any training document that shares one. It works, and it works only against copies.
question = ("A train leaves the station at half past two travelling at sixty "
"kilometres an hour. How far has it gone by ten past four?")
corpus = {
"verbatim copy": "Worked solutions. A train leaves the station at half past two "
"travelling at sixty kilometres an hour. How far has it gone by "
"ten past four? Answer: 100 km.",
"reworded": "A train departs the station at 14:30 doing sixty kilometres per "
"hour. What distance has it covered by 16:10? The answer is 100 km.",
"unrelated": "A bus leaves the depot at seven in the morning and returns at six.",
}
def ngrams(t, n):
w = t.lower().replace("?", "").replace(".", "").replace(":", " ").split()
return {tuple(w[i:i+n]) for i in range(len(w) - n + 1)}
q13 = ngrams(question, 13)
for name, doc in corpus.items():
hits = len(q13 & ngrams(doc, 13))
print(f"{name:15} 13-gram overlap {hits:2} {'LEAK' if hits else 'clean'}")
# verbatim copy 13-gram overlap 12 LEAK
# reworded 13-gram overlap 0 clean
# unrelated 13-gram overlap 0 cleanWhat to keep and what to skip#
- Keep: the three costs of a duplicate — compute, memorisation, and a mixture you no longer control.
- Keep: exact hashing catches mirrors, shingle similarity catches templates, and the threshold that works depends on document length.
- Keep: mixtures are chosen by training small proxy models and comparing, not by argument.
- Skip: the internals of MinHash and locality-sensitive hashing unless you are building the pipeline. Knowing that it approximates Jaccard cheaply is enough here; the unsupervised topic derives it if you want it.
- Skip: memorising which public corpus contains what. They are superseded every few months.
Where people go wrong
Attributing a model's strengths to its architecture when they come from its data. Two models with identical architectures and different corpora can differ enormously.
If this comes up in an interview
What matters most for pretraining quality?
Increasingly the data: deduplication, quality filtering and the mixture of sources. Architectures have largely converged, and much of the remaining gap between models comes from what they were trained on.
Why deduplicate a pretraining corpus?
Repeated text is memorised rather than generalised from, and it wastes compute on seeing the same thing many times. Near-duplicates matter as much as exact copies.
Build this: Train two tiny language models on the same size corpus, one deduplicated and filtered and one raw. Compare their loss and their samples.
Watch it explained
This is one section of Pretraining, 17 hours across 4 sections. Read it in full to get the whole topic in order.