Pretraining

Everything a large model can do comes from one task repeated at enormous scale: predict the next token. This section is about why that is enough to produce something that writes code and follows an argument, and about scaling laws, the empirical relationships between compute, data, model size and loss that decide how a training run is planned before a single GPU is booked. It also accounts for a good deal of model behaviour you would otherwise have to memorise: why knowledge stops at a date, why a model is fluent about things it never checked.

One objective, stated precisely#

The objective is worth stating exactly, because most confusion about model behaviour dissolves once you have. At every position in the text the model produces a probability for every token in its vocabulary — a hundred thousand numbers that sum to one. The loss at that position is the negative logarithm of the probability it gave to the token that actually came next. The training loss is that number averaged over every position in the corpus, and it is the only thing the run optimises. Nothing in it mentions truth, helpfulness, reasoning or the user.

The unit is worth knowing too, because it is what people mean when they quote a loss. Measured in nats, a loss of L corresponds to a perplexity of e to the power L, which reads as the number of equally likely tokens the model was effectively choosing between. A model with perplexity 4 is guessing between four options at each step; one with perplexity 1.01 has almost no uncertainty left. Here is the whole objective, at a scale you can read.

import math, collections

text = """the cat sat on the mat . the cat sat on the floor . the dog sat on the mat .
the dog ate the food . the cat ate the food . the cat sat on the chair ."""
tokens = text.split()
vocab = sorted(set(tokens))

# The objective, literally: for every position, a distribution over the vocabulary,
# scored by the log probability it gave to the token that actually came next.
counts = collections.defaultdict(collections.Counter)
for a, b in zip(tokens, tokens[1:]):
    counts[a][b] += 1

def predict(prev):
    c = counts[prev]
    total = sum(c.values())
    return {w: (c[w] + 1) / (total + len(vocab)) for w in vocab}   # add-one smoothing

loss = sum(-math.log(predict(a)[b]) for a, b in zip(tokens, tokens[1:]))
loss /= len(tokens) - 1

print(f"vocabulary {len(vocab)} words, {len(tokens)} tokens")
print("cross-entropy loss  " f"{loss:.3f} nats")
print("perplexity          " f"{math.exp(loss):.1f} words")
print("uniform guessing    " f"{len(vocab):.1f} words")

p = predict("the")
top = sorted(p.items(), key=lambda kv: -kv[1])[:4]
print("after 'the':", ", ".join(f"{w} {q:.2f}" for w, q in top))

# vocabulary 11 words, 40 tokens
# cross-entropy loss  1.513 nats
# perplexity          4.5 words
# uniform guessing    11.0 words
# after 'the': cat 0.22, dog 0.13, food 0.13, mat 0.13

That is the whole of pretraining. Everything else — the transformer, the optimiser, the cluster — is machinery for computing that average over trillions of positions and nudging a few billion numbers to make it smaller.

Why that one task produces general capability#

The usual explanation is that predicting text well requires modelling whatever produced the text, which is true and unsatisfying because it is not measurable. Here is the measurable version. Every regularity in the corpus is a discount on the loss. If some structure in the data — grammar, arithmetic, the fact that a name mentioned earlier tends to reappear — lets you assign more probability to what comes next, then a model that represents that structure scores lower loss than one that does not, and gradient descent will find it. Capability is not a goal of the objective; it is the cheapest way to satisfy it.

You can price one such regularity exactly. Take documents that mention an identifier and then repeat it later. A model with no way to use earlier context can only fall back on how often each identifier appears in the corpus overall. A model that can attend to what it already read just copies.

import math, random, collections

random.seed(0)
ids = [f"{random.randrange(10000):04d}" for _ in range(4000)]
docs = [["opened", i, ";", "wrote", "log", ";", "closed", i] for i in ids]

# Where in each document does the model have to produce the id again?
target = 7

# Model A: no memory of the document. It can only learn the marginal over ids.
marginal = collections.Counter(ids)
n = sum(marginal.values())
loss_a = sum(-math.log(marginal[d[target]] / n) for d in docs) / len(docs)

# Model B: allowed one rule, copy the id it already saw in this document.
loss_b = sum(-math.log(0.999 if d[target] == d[1] else 0.0001) for d in docs) / len(docs)

print(f"{len(docs)} documents, {len(set(ids))} distinct ids")
print("context-blind model   " f"{loss_a:.3f} nats at the id position")
print("copies from context   " f"{loss_b:.3f} nats at the id position")
print("the rule is worth     " f"{loss_a - loss_b:.3f} nats per document")

# 4000 documents, 3308 distinct ids
# context-blind model   8.042 nats at the id position
# copies from context   0.001 nats at the id position
# the rule is worth     8.041 nats per document

Now scale that argument up. Closing a bracket you opened four hundred tokens ago pays. Getting the tense right pays. Knowing that a function defined at the top of the file takes two arguments pays. Knowing that the capital of Peru is Lima pays. So does following an argument well enough to guess its conclusion. Each is a discount, the discounts compound, and none of them were asked for. What you must not conclude is that the model reasons the way you do; the honest claim is only that the cheapest way to predict this corpus turned out to include something that behaves like it much of the time, and behaves like nothing at all the rest.

Scaling laws#

Loss falls predictably as compute, data and parameters increase, following a power law across many orders of magnitude. This is why frontier labs can plan: given a compute budget, you can estimate the loss you will reach and how to split the budget between model size and data.

The version you can use is one line of arithmetic. Training compute is about six FLOPs per parameter per token — two for the forward pass and four for the backward — so a run costs roughly C equals 6 times N times D, where N is parameters and D is training tokens. Everything else is division: by the throughput of a GPU, by the fraction of that throughput you actually achieve, by the hourly price. Do it before you ask anyone for a cluster.

# C = 6 * N * D is the standard approximation: a forward and backward pass
# together cost about six FLOPs per parameter per token. Everything below is
# that one line and a division.
PEAK, MFU, RATE = 989e12, 0.40, 3.00      # H100 bf16 FLOP/s, realised fraction, $/GPU-hour

print("model     params    tokens      FLOPs      GPU-hours         cost   days")
for name, params, per_param, gpus in [
    ("small",  1.5e9,  20,    8),
    ("7B",       7e9,  20,   64),
    ("7B x10",   7e9, 200,   64),
    ("70B",      7e10, 20,  512),
    ("400B",     4e11, 20, 4096),
]:
    tokens = params * per_param
    flops = 6 * params * tokens
    gpu_hours = flops / (PEAK * MFU) / 3600
    days = gpu_hours / gpus / 24
    print(f"{name:8} {params/1e9:6.1f} B {tokens/1e12:7.2f} T  {flops:9.2e} "
          f"{gpu_hours:11,.0f}  " "$" f"{gpu_hours*RATE:11,.0f} {days:6.1f}")
print("(days is wall clock on 8, 64, 64, 512 and 4096 GPUs respectively)")

# model     params    tokens      FLOPs      GPU-hours         cost   days
# small       1.5 B    0.03 T   2.70e+20         190  $        569    1.0
# 7B          7.0 B    0.14 T   5.88e+21       4,129  $     12,386    2.7
# 7B x10      7.0 B    1.40 T   5.88e+22      41,287  $    123,862   26.9
# 70B        70.0 B    1.40 T   5.88e+23     412,875  $  1,238,625   33.6
# 400B      400.0 B    8.00 T   1.92e+25  13,481,631  $ 40,444,894  137.1
# (days is wall clock on 8, 64, 64, 512 and 4096 GPUs respectively)

Three things fall out of that table. A compute-optimal 7B model is a four-thousand-GPU-hour job — about twelve thousand dollars, which is a team's monthly cloud bill rather than a moonshot. Training the same 7B on ten times the data costs ten times as much and is exactly what production models now do, because the extra training is paid once and the smaller model is served forever. And the last row is why the list of organisations doing frontier pretraining is short: forty million dollars of compute, four thousand GPUs held for four and a half months, and that is the successful run, not the failed ones before it.

The other half of scaling is how to split a fixed budget between a bigger model and more data, and C equals 6ND forces the trade: every parameter you add is tokens you can no longer afford. The compute-optimal answer, from the Chinchilla work, is around twenty training tokens per parameter — a ratio, not a number of tokens, so it moves with the budget.

# One budget, five ways to spend it. C = 6ND fixes the trade: every parameter
# you add is tokens you cannot afford.
C = 1e22                                   # FLOPs, roughly two days on 64 H100s

print("params    tokens   tokens/param   verdict")
for params in [1e9, 5e9, 8.3e9, 20e9, 70e9]:
    tokens = C / (6 * params)
    ratio = tokens / params
    if ratio > 60:
        verdict = "small model, drowning in data"
    elif ratio > 8:
        verdict = "near the compute-optimal band"
    else:
        verdict = "big model, badly undertrained"
    print(f"{params/1e9:5.1f} B  {tokens/1e9:7.1f} B  {ratio:11.1f}   {verdict}")

opt = (C / (6 * 20)) ** 0.5                # the size where tokens/param lands on 20
print()
print(f"compute-optimal size at this budget: {opt/1e9:.1f} B parameters")

# params    tokens   tokens/param   verdict
#   1.0 B   1666.7 B       1666.7   small model, drowning in data
#   5.0 B    333.3 B         66.7   small model, drowning in data
#   8.3 B    200.8 B         24.2   near the compute-optimal band
#  20.0 B     83.3 B          4.2   big model, badly undertrained
#  70.0 B     23.8 B          0.3   big model, badly undertrained
#
# compute-optimal size at this budget: 9.1 B parameters

Two caveats keep this honest. Compute-optimal minimises training loss for a training budget, and says nothing about serving cost, which is why almost every model you can actually call is deliberately overtrained — a smaller model pushed well past twenty tokens per parameter is cheaper for every request it ever answers. And at large budgets the recommended token count exceeds the high-quality text that exists, at which point the binding constraint stops being compute and becomes data, which is the subject of the next section.

What this explains#

• Knowledge cutoffs: the model knows the corpus, so it does not know what happened afterwards.
• Confident errors: it was trained to produce plausible continuations, not to signal uncertainty. Fluency and correctness were never separated.
• Uneven competence: it is strong where the corpus was rich and weak where it was thin, which does not track human intuitions about difficulty.
• Format sensitivity: text laid out like its training data is easier to continue well.

Understanding pretraining explains model behaviour better than any amount of prompting folklore, which is the practical reason to learn it even if you will never run one.

Emergence is contested#

What to keep and what to skip#

• Keep: the objective in one sentence, that perplexity is the exponential of the loss, and C equals 6ND. Those three let you answer most of what is asked here.
• Keep: roughly twenty tokens per parameter as the compute-optimal ratio, and that production models are deliberately overtrained past it because inference is paid forever.
• Skip: the exact coefficients of any published scaling law. They are fitted to one lab's setup, they are superseded regularly, and nobody is asking you to recite them.
• Skip: arguing about emergence. Say it is reported, say the metric critique exists, and move on.
• Skip: memorising GPU prices. Memorise the shape of the calculation and look the rate up.

Architectures have largely converged; corpora have not. Most of what separates a strong model from a weak one is what went into it — what was collected, what was thrown out, and in what proportions. This section covers deduplication and filtering, the mixture question of how much code, how much maths and how many languages, and contamination, where benchmark data leaks into training and quietly voids your evaluation. It gets discussed less than architecture because it is less publishable and more commercially sensitive, not because it matters less.

The quiet majority of the work#

Architectures have converged; data has not. Much of what separates strong models from weak ones is the corpus: what was included, what was removed, and how it was balanced. This is less discussed than architecture because it is less publishable and more commercially sensitive.

There is a reason data wins that has nothing to do with fashion, and it is worth being precise about it. Compute for a training run is roughly six FLOPs per parameter per token, so the cost is linear in the number of tokens you train on. A filter that reaches the same loss on half the tokens has therefore halved the cost of the run — it is worth exactly as much as doubling the size of your cluster, and it costs a few hundred CPU-hours rather than a few million dollars. Architecture changes move the loss by a few per cent. Data changes move how many tokens you need, which moves the whole budget.

The other half of the argument is that data decides what the model is good at, which architecture does not. A better attention variant makes the same model slightly better at everything. Adding a hundred billion tokens of code makes it much better at code and, awkwardly for anyone hoping for a clean story, somewhat better at reasoning tasks that have nothing to do with programming.

Deduplication, and what duplicates actually cost#

The web repeats itself. Mirrored sites, syndicated articles, template pages, quotations, and the same licence text on ten million pages. Left in, a document that appears a thousand times gets a thousand times the influence on the weights, and there are three separate costs to that. Compute is spent seeing the same thing again. The model memorises rather than generalises, because reproducing an exact string it has seen a thousand times is the loss-minimising move. And the mixture you carefully chose is silently overwritten by whatever the web happens to copy most.

All three are measurable at a scale that fits on one machine. Take five hundred ordinary sentences and one boilerplate page mirrored two hundred times, fit the same counting model with and without the duplicates, and score it on the boilerplate and on a sentence it has never seen.

import math, random, collections

random.seed(1)
subj = ["the report", "the server", "the invoice", "the driver", "the parcel"]
verb = ["failed", "arrived", "expired", "returned", "stalled"]
when = ["on monday", "at noon", "last week", "overnight", "in transit"]

def sentence():
    return f"{random.choice(subj)} {random.choice(verb)} {random.choice(when)} ."

unique = [sentence() for _ in range(500)]
boiler = "this page requires javascript to display correctly ."
corpus = unique + [boiler] * 200          # one boilerplate page, mirrored 200 times

def fit(docs):
    c = collections.defaultdict(collections.Counter)
    for d in docs:
        t = ["<s>"] + d.split()
        for a, b in zip(t, t[1:]):
            c[a][b] += 1
    return c

def loss(model, doc, vocab):
    t = ["<s>"] + doc.split()
    total = 0.0
    for a, b in zip(t, t[1:]):
        c = model[a]
        total += -math.log((c[b] + 1) / (sum(c.values()) + vocab))
    return total / (len(t) - 1)

vocab = len(set(" ".join(corpus).split())) + 1
with_dupes = fit(corpus)
deduped = fit(unique + [boiler])

held_out = sentence()
print(f"corpus {len(corpus)} documents, {sum(len(d.split()) for d in corpus)} tokens")
print(f"share of tokens that are the one mirrored page: "
      f"{200*len(boiler.split())/sum(len(d.split()) for d in corpus):.1%}")
print("                     boilerplate   held-out sentence")
print(f"with duplicates      {loss(with_dupes, boiler, vocab):11.3f} {loss(with_dupes, held_out, vocab):18.3f}")
print(f"deduplicated         {loss(deduped, boiler, vocab):11.3f} {loss(deduped, held_out, vocab):18.3f}")

# corpus 700 documents, 4488 tokens
# share of tokens that are the one mirrored page: 35.7%
#                      boilerplate   held-out sentence
# with duplicates            0.275              1.136
# deduplicated               3.067              1.072

Finding the duplicates is where it gets awkward. Hashing catches exact copies and nothing else, and the web's duplication is mostly not exact: the same template with a different order number, the same article with a different date line. The standard answer is to compare sets of overlapping word sequences, called shingles, and treat two documents as near-duplicates when their Jaccard similarity passes a threshold, with MinHash used to make that comparison affordable across billions of documents.

import hashlib

A = ("Order 41823 shipped on 4 March. Track your parcel using the reference above. "
     "Delivery is normally two working days. Contact support if it has not arrived.")
B = A.replace("41823", "41824").replace("4 March", "5 March")   # same template, new order
C = A                                                            # a true mirror
D = ("The harbour was rebuilt in 1892 after a storm destroyed the old wooden pier. "
     "Records show the council borrowed heavily to pay for the stone replacement.")
pages = {"A": A, "B": B, "C": C, "D": D}

seen, removed = set(), []
for name, p in pages.items():
    h = hashlib.sha256(p.encode()).hexdigest()
    if h in seen:
        removed.append(name)
    seen.add(h)
print("exact hashing removes:", removed, "- B survives, and B is a template")

def shingles(t, k):
    w = t.lower().replace(".", "").split()
    return {tuple(w[i:i+k]) for i in range(len(w) - k + 1)}

print()
print("Jaccard of A against B, by shingle size")
for k in [3, 5, 8, 13]:
    a, b = shingles(A, k), shingles(B, k)
    j = len(a & b) / len(a | b)
    print(f"  k={k:2}   {j:.2f}   {'caught at 0.8' if j >= 0.8 else 'slips through at 0.8'}")

# exact hashing removes: ['C'] - B survives, and B is a template
#
# Jaccard of A against B, by shingle size
#   k= 3   0.66   slips through at 0.8
#   k= 5   0.63   slips through at 0.8
#   k= 8   0.58   slips through at 0.8
#   k=13   0.47   slips through at 0.8

That is the part worth understanding rather than memorising, because it decides whether your deduplication does anything. Near-duplicate detection is a ratio between the shared part and the varying part, so the same two-token difference is invisible in a long page and fatal in a short one.

SHARED = [
    "Track your parcel using the reference above.",
    "Delivery is normally two working days.",
    "Contact support if the parcel has not arrived.",
    "Returns are free within thirty days of delivery.",
    "This message was sent automatically, please do not reply.",
    "Our opening hours are nine until five on weekdays.",
    "Gift receipts can be requested from the orders page.",
    "Packaging is recycled wherever local facilities allow it.",
]

def shingles(t, k=5):
    w = t.lower().replace(".", "").replace(",", "").split()
    return {tuple(w[i:i+k]) for i in range(len(w) - k + 1)}

print("shared lines   words   Jaccard(A,B)   caught at 0.8")
for n in [1, 2, 4, 8]:
    tail = " ".join(SHARED[:n])
    a = "Order 41823 shipped on 4 March. " + tail
    b = "Order 41824 shipped on 5 March. " + tail
    j = len(shingles(a) & shingles(b)) / len(shingles(a) | shingles(b))
    print(f"{n:12}   {len(a.split()):5}   {j:12.2f}   {'yes' if j >= 0.8 else 'no'}")

# shared lines   words   Jaccard(A,B)   caught at 0.8
#            1      13           0.29   no
#            2      19           0.50   no
#            4      35           0.72   no
#            8      70           0.86   yes

Quality filtering#

After deduplication the corpus is still mostly not writing. It is navigation bars, cookie banners, keyword spam, tables of numbers, autogenerated listings and pages that are one sentence and a link. The first pass is cheap heuristics, each one a single statistic with a threshold, and it is worth writing them out because it makes the shape of the problem obvious.

docs = {
    "article":   "The harbour was rebuilt in 1892 after a storm destroyed the old "
                 "wooden pier. Records from the period show that the town council "
                 "borrowed heavily to pay for the stone replacement, and the debt "
                 "was not cleared until after the war. The new pier was longer than "
                 "the one it replaced, and the fishing fleet grew by a third in the "
                 "decade that followed. A plaque at the landward end still lists the "
                 "names of the councillors who signed for the loan, and the date of "
                 "the storm that made it necessary.",
    "navigation": "Home | Products | About | Contact | Login | Register | Terms | "
                  "Privacy | Cookies | Home | Products | About | Contact | Home | "
                  "Products | About | Contact | Login | Register | Terms | Privacy",
    "keyword spam": "cheap flights cheap flights cheap flights book cheap flights "
                    "cheap flights cheap flights cheap flights now cheap flights "
                    "cheap flights book cheap flights cheap flights cheap flights "
                    "cheap flights cheap flights cheap flights cheap flights now",
    "table":     "2019 4.1 3.9 | 2020 4.4 4.0 | 2021 4.8 4.2 | 2022 5.1 4.6 | "
                 "2023 5.5 4.9 | 2024 5.9 5.1 | 2025 6.2 5.4 | 2026 6.6 5.8 | "
                 "2027 7.0 6.1 | 2028 7.4 6.4 | 2029 7.8 6.7 | 2030 8.2 7.0 | "
                 "2031 8.6 7.3 | 2032 9.0 7.6 | 2033 9.4 7.9 | 2034 9.8 8.2",
    "truncated": "Click here to continue reading",
}

STOP = {"the", "and", "of", "to", "in", "a", "was", "after", "that", "it", "at"}

def flags(t):
    words = t.split()
    alpha = sum(c.isalpha() or c.isspace() for c in t) / len(t)
    stop = sum(w.lower().strip(".,") in STOP for w in words) / len(words)
    uniq = len(set(w.lower() for w in words)) / len(words)
    return {"too short": len(words) < 50, "symbol heavy": alpha < 0.85,
            "few stop words": stop < 0.10, "repetitive": uniq < 0.5}

names = list(flags(docs["article"]))
print(f"{'document':13} {'words':>5} " + " ".join(f"{k:>15}" for k in names))
for name, t in docs.items():
    f = flags(t)
    print(f"{name:13} {len(t.split()):5} " +
          " ".join(f"{('DROP' if v else '-'):>15}" for v in f.values()))

print()
print("length threshold sweep: how many of the 5 documents survive it alone")
for k in [10, 50, 100, 200]:
    kept = [n for n, t in docs.items() if len(t.split()) >= k]
    print(f"  >= {k:3} words   keeps {len(kept)}   {', '.join(kept) or 'nothing'}")

# document      words       too short    symbol heavy  few stop words      repetitive
# article          92               -               -               -               -
# navigation       41            DROP               -            DROP            DROP
# keyword spam     36            DROP               -            DROP            DROP
# table            63               -            DROP            DROP               -
# truncated         5            DROP               -               -               -
#
# length threshold sweep: how many of the 5 documents survive it alone
#   >=  10 words   keeps 4   article, navigation, keyword spam, table
#   >=  50 words   keeps 2   article, table
#   >= 100 words   keeps 0   nothing
#   >= 200 words   keeps 0   nothing

The sweep at the bottom is the part that costs people a corpus. A length threshold is the cheapest filter and the most destructive one, because it is the only filter that discards good text in proportion to how much of it is short. Move it from fifty words to a hundred here and everything is gone. On a real crawl the same move quietly removes most reference material, most question-and-answer pages and most code comments, and the loss curve will not tell you, because the run still trains perfectly well on what is left.

The second pass is a learned classifier. Label a sample of pages for whether they look like something worth learning from — public work such as FineWeb used a larger model to produce those labels — train a small, fast classifier on the labels, and score the whole corpus with it. The reported result is the one that matters for planning: keeping the pages a classifier judges educational gave a large improvement on knowledge and reasoning benchmarks at the same number of training tokens. Same compute, same architecture, better model.

Safety and legal filtering#

A third pass removes what should not be learned regardless of how well written it is: personal information such as names beside addresses, phone numbers, identity numbers and credentials; material that is illegal to hold; and content the model should not become fluent in producing. It is the least interesting stage technically and the one with the most consequences, because anything left in is memorised and can be reproduced, and removing it after training is not possible.

• Personal data: pattern matching for the structured kinds, classifiers for the rest, and accepting that recall will not be one.
• Credentials and keys: scraped code is full of them, and a model that has memorised a live API key is a liability rather than a curiosity.
• Licensing and provenance: what you are permitted to train on is a legal question with live disagreement, and the answer determines whether a model can be released at all.
• Harmful content: removed at this stage as well as being addressed in post-training, because a capability never learned is more reliably absent than one trained against afterwards.

Mixture#

How much code, how much maths, how many languages, how much conversation. The proportions shape capability directly. Adding more code improves reasoning in ways that extend beyond programming, which is one of the more surprising findings.

There is no theory for choosing them. The method everyone uses is to train several small proxy models, one per candidate mixture, evaluate them on the tasks you care about and scale up the winner. It works because the ranking of mixtures is reasonably stable across scale even though the absolute scores are not, and it is affordable because the proxy models are a thousandth of the size. If asked how you would pick a mixture, that answer — train small, compare, scale the winner — is the one being looked for, and 'principled reasoning about proportions' is not.

Contamination#

The standard defence is n-gram matching: take every benchmark question, cut it into overlapping sequences of thirteen words, and drop any training document that shares one. It works, and it works only against copies.

question = ("A train leaves the station at half past two travelling at sixty "
            "kilometres an hour. How far has it gone by ten past four?")

corpus = {
  "verbatim copy": "Worked solutions. A train leaves the station at half past two "
                   "travelling at sixty kilometres an hour. How far has it gone by "
                   "ten past four? Answer: 100 km.",
  "reworded":      "A train departs the station at 14:30 doing sixty kilometres per "
                   "hour. What distance has it covered by 16:10? The answer is 100 km.",
  "unrelated":     "A bus leaves the depot at seven in the morning and returns at six.",
}

def ngrams(t, n):
    w = t.lower().replace("?", "").replace(".", "").replace(":", " ").split()
    return {tuple(w[i:i+n]) for i in range(len(w) - n + 1)}

q13 = ngrams(question, 13)
for name, doc in corpus.items():
    hits = len(q13 & ngrams(doc, 13))
    print(f"{name:15} 13-gram overlap {hits:2}   {'LEAK' if hits else 'clean'}")

# verbatim copy   13-gram overlap 12   LEAK
# reworded        13-gram overlap  0   clean
# unrelated       13-gram overlap  0   clean

What to keep and what to skip#

• Keep: the three costs of a duplicate — compute, memorisation, and a mixture you no longer control.
• Keep: exact hashing catches mirrors, shingle similarity catches templates, and the threshold that works depends on document length.
• Keep: mixtures are chosen by training small proxy models and comparing, not by argument.
• Skip: the internals of MinHash and locality-sensitive hashing unless you are building the pipeline. Knowing that it approximates Jaccard cheaply is enough here; the unsupervised topic derives it if you want it.
• Skip: memorising which public corpus contains what. They are superseded every few months.

A base model is the output of a data pipeline far more than of an architecture. This section follows the whole thing: web crawls turned into clean text, the filters and quality classifiers that throw most of it away, deduplication, training the tokenizer, and then the run itself and what gets watched while it goes. You finish by running a miniature version, which makes every stage concrete at a scale that fits on one machine.

The stages#

web crawl snapshots
  -> extract main text from HTML            (drop menus, ads, boilerplate)
  -> language identification                (keep the languages you want)
  -> heuristic filters                      (length, symbol ratios, repeated lines)
  -> quality classifier                     (keep text that looks like good writing or teaching)
  -> deduplication                          (exact and near-duplicate removal)
  -> remove personal data and unsafe content
  -> mix with code, books, papers, maths    (the mixture is tuned)
  -> tokenize
  -> train

Quality filtering#

Public datasets such as FineWeb documented this carefully. One notable finding: training a small classifier to keep pages with educational value, using labels produced by a larger model, gave a large improvement on knowledge and reasoning benchmarks for the same number of training tokens. Data quality moves results as much as model size.

Deduplication#

The web repeats itself: mirrored sites, templates, copied articles. Duplicates waste compute and make models memorise. Near-duplicate detection with MinHash, the same technique as in the unsupervised topic, runs across the whole corpus.

Training the tokenizer#

The model never sees text. It sees integers, and the tokenizer is the thing that decides which integers. Byte pair encoding builds that mapping from the corpus itself rather than from a dictionary: start with the 256 possible bytes, count every adjacent pair, merge the most frequent one into a new symbol, and repeat. Each merge adds one entry to the vocabulary. Run it ten thousand times on English and you get whole common words; run it on code and you get four-space indents and double colons.

That is the whole algorithm, and it is short enough to write. The number to watch is bytes per token, because it is the compression ratio that decides how much text a sequence of a given length can hold.

import collections, random

random.seed(0)
WORDS = ("the of and to in a is that it was for on with as by at from but not are "
         "this have has were they their which you one all would there we been when "
         "who will more if out so what about into them then some could time other "
         "than only its over also new because any these two may first well after "
         "council harbour pier storm fishing fleet replacement borrowed destroyed "
         "rebuilt reopened records period debt cleared stone wooden plaque").split()
weights = [1 / (i + 1) for i in range(len(WORDS))]          # a Zipf-ish frequency curve
corpus = " ".join(random.choices(WORDS, weights, k=12000))
held_out = " ".join(random.choices(WORDS, weights, k=2000))

def train_bpe(text, merges):
    seq, rules, nxt = list(text.encode()), [], 256
    for _ in range(merges):
        (a, b), n = collections.Counter(zip(seq, seq[1:])).most_common(1)[0]
        if n < 2:
            break
        out, i = [], 0
        while i < len(seq):
            if i + 1 < len(seq) and seq[i] == a and seq[i + 1] == b:
                out.append(nxt); i += 2
            else:
                out.append(seq[i]); i += 1
        seq, rules, nxt = out, rules + [((a, b), nxt)], nxt + 1
    return rules

def encode(text, rules):
    seq = list(text.encode())
    for (a, b), new in rules:
        out, i = [], 0
        while i < len(seq):
            if i + 1 < len(seq) and seq[i] == a and seq[i + 1] == b:
                out.append(new); i += 2
            else:
                out.append(seq[i]); i += 1
        seq = out
    return seq

def spell(tok, rules):
    table = {new: (a, b) for (a, b), new in rules}
    if tok < 256:
        return chr(tok)
    a, b = table[tok]
    return spell(a, rules) + spell(b, rules)

rules = train_bpe(corpus, 1200)
print("first merges learned:", " ".join(repr(spell(n, rules)) for _, n in rules[:8]))
print()
print("vocab   tokens for 2000 held-out words   bytes/token")
for k in [0, 100, 300, 600, 1200]:
    sub = rules[:k]
    n = len(encode(held_out, sub))
    print(f"{256 + len(sub):5}   {n:31}   {len(held_out.encode())/n:11.2f}")

# first merges learned: ' t' ' th' ' the' ' a' ' o' ' i' ' of' ' w'
#
# vocab   tokens for 2000 held-out words   bytes/token
#   256                              8158          1.00
#   356                              2476          3.29
#   556                              1553          5.25
#   856                              1392          5.86
#  1368                              1313          6.21

The trade behind that curve is worth stating, because it is asked. A larger vocabulary means shorter sequences, which means less compute per document and more text inside a fixed context window. It also means a larger embedding matrix and a larger output projection, and rarer tokens that each get seen fewer times during training. Somewhere in the middle those cross, and the crossing point has moved up as models have grown.

The failure here is specific and permanent, which is what makes it worth a paragraph. The tokenizer is trained once, on a sample of the corpus, and then frozen for the life of the model — you cannot change it later without retraining everything. So the mixture of that sample decides, forever, how efficiently the model can represent anything. A tokenizer fitted mostly to English spends two to four times as many tokens on the same content in a language it saw little of, and every one of those tokens is paid again at training time, at inference time, and against the context window. Measure bytes per token on the kinds of text you actually care about before you freeze it.

The run#

• A tokenizer is trained on a sample of the corpus first, with byte pair encoding.
• The model trains on trillions of tokens with a warmup and a decaying learning rate.
• Teams watch the loss curve for spikes, gradient norms, and small benchmark evaluations every few thousand steps.
• Loss spikes are handled by rewinding to an earlier checkpoint, sometimes skipping the data that caused it.
• Near the end, higher-quality data is often weighted more heavily, a stage sometimes called annealing or midtraining.

What actually goes wrong in a long run#

A pretraining run is a month-long single process, and the interesting failures are not crashes. A crash you notice. The failure that costs a team a week is a loss spike: the curve jumps by orders of magnitude for one step and then either settles back or never recovers. The causes are mundane — a shard that escaped the filters, a batch of near-identical documents, an unlucky interaction between a large gradient and the optimiser's accumulated state — and the reason one bad batch can do permanent damage is that Adam scales updates by the running estimate of gradient magnitude, so a single enormous gradient moves the weights a long way and poisons that estimate for thousands of steps afterwards.

The mechanism is small enough to reproduce on a laptop. Train anything for a hundred steps, hand it one batch whose inputs are five hundred times too large — a shard that arrived unnormalised, which is an ordinary pipeline bug — and watch what the defences do.

import torch, torch.nn as nn

def run(clip=None, skip=False, steps=120, bad_step=60):
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(16, 64), nn.ReLU(), nn.Linear(64, 1))
    opt = torch.optim.Adam(model.parameters(), lr=1e-2)
    w = torch.randn(16, 1)
    history = []
    for step in range(steps):
        g = torch.Generator().manual_seed(step)
        x = torch.randn(32, 16, generator=g)
        if step == bad_step:
            x = x * 500                      # one shard arrived unnormalised
        y = x @ w
        loss = ((model(x) - y) ** 2).mean()
        if skip and history and loss.item() > 20 * sorted(history[-20:])[len(history[-20:]) // 2]:
            history.append(history[-1])      # refuse the batch, keep the weights
            continue
        opt.zero_grad()
        loss.backward()
        if clip:
            nn.utils.clip_grad_norm_(model.parameters(), clip)
        opt.step()
        history.append(loss.item())
    return history

for label, kw in [("no defence", {}), ("clip at 1.0", {"clip": 1.0}),
                  ("clip + skip", {"clip": 1.0, "skip": True})]:
    h = run(**kw)
    print(f"{label:12} step 59 {h[59]:9.3f}   step 60 {h[60]:12.1f}   "
          f"step 70 {h[70]:10.3f}   step 119 {h[119]:9.3f}")

# no defence   step 59     0.136   step 60     147602.5   step 70      5.689   step 119    12.254
# clip at 1.0  step 59     0.113   step 60     180806.3   step 70      0.029   step 119     0.034
# clip + skip  step 59     0.113   step 60          0.1   step 70      0.030   step 119     0.036

Two things in that output are worth separating. Clipping does not stop the loss being reported as 180,806 on the bad batch — the batch really is that wrong — it stops the update from being proportionally enormous, and the run is back under 0.03 within ten steps. Skipping goes further and refuses the batch outright when its loss is far above the recent median, so the spike never appears at all. In a real run you want both, because clipping is a bound on damage and skipping is a bound on how often you pay it.

That gives the standard playbook, which is what an interviewer is after when they ask what you do about a loss spike. Gradient clipping is always on, not switched on after a problem. Checkpoints are frequent, because the recovery is to rewind. When a spike leaves permanent damage you restart from the last good checkpoint and skip forward past the batches that caused it, and if the spike reproduces at the same point in the data, the problem is the data and not the optimiser. Checkpoint frequency is a straight cost calculation: four thousand and ninety-six GPUs at three dollars an hour is about 295,000 dollars a day, so an hour of lost progress is roughly 12,000 dollars and writing a checkpoint every thirty minutes is cheap insurance.

Who should ever do this#

Almost nobody, and being clear about that is more useful than any of the detail above. The reason to learn the pipeline is that it explains model behaviour and it comes up in interviews, not because you are going to run it. The honest ladder looks like this, and the right answer is nearly always as high up it as you can get away with.

approach                 typical cost     changes          right when
prompting                pennies          nothing          the capability is already there
retrieval                a day's work     what it sees     the gap is knowledge, not behaviour
LoRA fine-tune           tens of dollars  behaviour        format, tone, a narrow task
full fine-tune           hundreds         behaviour        LoRA is not enough, rare
continued pretraining    thousands +      knowledge too    a domain genuinely absent from the corpus
pretraining from scratch millions         everything       you need control of every token

There are genuinely three cases for the bottom rung. A domain whose text is not in any public corpus and cannot be — proprietary formats, instrument traces, a language with almost no web presence. A hard requirement to account for the provenance and licence of every training token, which no public base model can give you. And research on the training process itself, where the run is the experiment. Wanting the model to know your documents is not on that list; that is retrieval, and the section on fine-tuning against RAG argues it out.

Continued pretraining is the rung people overlook, and it is often the right one. You take a released base model and keep training it on your own corpus at a low learning rate, usually mixed with general text so it does not forget everything else. It costs thousands rather than millions because you are buying the last few per cent of the token budget rather than all of it, and it is the realistic answer for a domain the base model has seen a little of and not enough.

Run a miniature#

Projects such as nanochat and nanoGPT let you run the whole pipeline at small scale: train a tokenizer, pretrain a small model on a few billion tokens from a public dataset, then fine-tune it to chat. Doing it once turns every paragraph above into something you have seen.

What to keep and what to skip#

• Keep: the order of the stages, and that most of them throw data away rather than add anything.
• Keep: byte pair encoding as repeated merging of the most frequent pair, and that the tokenizer is frozen for the life of the model.
• Keep: clip, checkpoint, rewind and skip, in that order, as the answer to a loss spike.
• Keep: the ladder, and that continued pretraining exists between fine-tuning and starting over.
• Skip: learning-rate schedule folklore. Warmup then cosine decay is the default and the variations are not what you are being asked about.
• Skip: implementing MinHash or a distributed deduplication job unless that is the role you are interviewing for.

Every model release comes with a report, and they are marketing documents as much as technical ones. This section is about reading one for what it actually establishes: which benchmarks were run under which conditions against which competitors, what a context length claim does and does not promise, and what the licence lets you do with the model and its outputs. What has been left out matters as much as what is there, so it lists the omissions worth noticing.

A checklist#

• Size, architecture, context length and tokenizer.
• Training data: sources, size in tokens, cut-off date, languages.
• Evaluation: which benchmarks, with what prompts and settings, compared against which models under the same conditions.
• Any statement about contamination checks.
• Known limitations and safety evaluations.
• Licence: can you use it commercially, fine-tune it, use its outputs to train other models?

Claims to check#

Comparison tables are often assembled from different sources with different settings. A 'context length' is the maximum the model accepts, not the length at which it uses information reliably; look for long-context retrieval evaluations. And a number without the prompt format and sampling settings cannot be reproduced.

Is the gap real?#

The single most useful habit when reading a results table is to work out the noise floor before reading the numbers. A benchmark score is a proportion measured on a finite sample, so it carries sampling error like any other proportion: about 1.96 times the square root of p times one minus p over n for a ninety-five per cent interval, and about 1.4 times that again for the gap between two models. Most published gaps are smaller than the interval around them.

import math

def half_width(p, n):
    return 1.96 * math.sqrt(p * (1 - p) / n)          # 95% interval on one score

def gap_needed(p, n):
    return 1.96 * math.sqrt(2 * p * (1 - p) / n)      # smallest gap that is not noise

print("items    one score is +/-   a real gap must exceed")
for n in [100, 250, 500, 1000, 5000, 20000]:
    print(f"{n:5}    {half_width(0.70, n)*100:12.1f} pts   {gap_needed(0.70, n)*100:16.1f} pts")

print()
for n, a, b in [(500, 0.712, 0.704), (500, 0.760, 0.700), (14000, 0.712, 0.704)]:
    gap = (a - b) * 100
    need = gap_needed(0.70, n) * 100
    print(f"{a:.1%} vs {b:.1%} on {n:6} items: gap {gap:.1f} pts, noise floor {need:.1f} pts"
          f"  -> {'real' if gap > need else 'indistinguishable'}")

print()
for gap in [0.008, 0.02, 0.05]:
    n = 1.96 ** 2 * 2 * 0.70 * 0.30 / gap ** 2
    print(f"to call a {gap*100:.1f} point gap real you need about {n:,.0f} items")

# items    one score is +/-   a real gap must exceed
#   100             9.0 pts               12.7 pts
#   250             5.7 pts                8.0 pts
#   500             4.0 pts                5.7 pts
#  1000             2.8 pts                4.0 pts
#  5000             1.3 pts                1.8 pts
# 20000             0.6 pts                0.9 pts
#
# 71.2% vs 70.4% on    500 items: gap 0.8 pts, noise floor 5.7 pts  -> indistinguishable
# 76.0% vs 70.0% on    500 items: gap 6.0 pts, noise floor 5.7 pts  -> real
# 71.2% vs 70.4% on  14000 items: gap 0.8 pts, noise floor 1.1 pts  -> indistinguishable
#
# to call a 0.8 point gap real you need about 25,210 items
# to call a 2.0 point gap real you need about 4,034 items
# to call a 5.0 point gap real you need about 645 items

Two honest qualifications, because someone will raise them. Scoring both models on the same items is a paired comparison and has lower variance than the formula above, so the true threshold is somewhat friendlier than 5.7 points on five hundred items — but you need the per-item results to compute it, and reports almost never publish them. And benchmark items are not independent draws from anything: they were written by people, in groups, about a fixed set of topics. The arithmetic is a floor on the uncertainty, not a complete account of it.

What a context length claim costs#

A long context number is a capability claim and a budget claim at once, and the budget half is arithmetic you can do while reading the announcement. Multiply the tokens you would actually put in the window by the input price.

PRICE_IN = 3.00 / 1e6          # dollars per input token, a typical mid-range rate

print("context used   cost of one call   cost of 1,000 calls a day, per month")
for tokens in [4_000, 32_000, 128_000, 200_000, 1_000_000]:
    c = tokens * PRICE_IN
    print(f"{tokens:12,}   " "$" f"{c:15.2f}   " "$" f"{c*1000*30:26,.0f}")

# context used   cost of one call   cost of 1,000 calls a day, per month
#        4,000   $           0.01   $                       360
#       32,000   $           0.10   $                     2,880
#      128,000   $           0.38   $                    11,520
#      200,000   $           0.60   $                    18,000
#    1,000,000   $           3.00   $                    90,000

Licence#

The licence section decides whether the model is usable at all, and it is the part of a release that most readers skip. Four clauses are worth finding every time. Whether commercial use is permitted at all, and whether it is capped by the licensee's user count or revenue. Whether you may fine-tune and redistribute the result. Whether outputs may be used to train other models, which several licences forbid outright and which quietly rules out distillation. And whether there is an acceptable-use policy attached that binds you downstream, because those are usually a separate document from the licence itself and are updated without a new model release.

What is usually missing#

Detailed data composition, exact compute, failed experiments and many negative results are often left out for competitive reasons. Note the gaps, and treat claims that depend on them with appropriate caution.

One omission is worth naming separately because it changes how you read everything else: per-item results. Without them you cannot compute a paired comparison, you cannot see which subset of the benchmark the gain came from, and you cannot tell a model that is broadly two points better from one that is thirty points better on one category and unchanged elsewhere. Those are completely different products, and the headline number is identical.

What to keep and what to skip#

• Keep: compute the noise floor before reading the table. On a few hundred items it is several points wide.
• Keep: a context length is what the model accepts, not what it uses well, and filling it is a monthly bill.
• Keep: the four licence clauses — commercial use, redistribution of fine-tunes, training on outputs, and the acceptable-use policy.
• Skip: the architecture diagram. It is nearly always the same architecture, and the differences rarely affect how you use it.
• Skip: reproducing their benchmark numbers. Run your own tasks instead; that is the only comparison that decides anything for you.