The pretraining pipeline, end to end

A base model is the output of a data pipeline far more than of an architecture. This section follows the whole thing: web crawls turned into clean text, the filters and quality classifiers that throw most of it away, deduplication, training the tokenizer, and then the run itself and what gets watched while it goes. You finish by running a miniature version, which makes every stage concrete at a scale that fits on one machine.

The stages#

web crawl snapshots
  -> extract main text from HTML            (drop menus, ads, boilerplate)
  -> language identification                (keep the languages you want)
  -> heuristic filters                      (length, symbol ratios, repeated lines)
  -> quality classifier                     (keep text that looks like good writing or teaching)
  -> deduplication                          (exact and near-duplicate removal)
  -> remove personal data and unsafe content
  -> mix with code, books, papers, maths    (the mixture is tuned)
  -> tokenize
  -> train

Quality filtering#

Public datasets such as FineWeb documented this carefully. One notable finding: training a small classifier to keep pages with educational value, using labels produced by a larger model, gave a large improvement on knowledge and reasoning benchmarks for the same number of training tokens. Data quality moves results as much as model size.

Deduplication#

The web repeats itself: mirrored sites, templates, copied articles. Duplicates waste compute and make models memorise. Near-duplicate detection with MinHash, the same technique as in the unsupervised topic, runs across the whole corpus.

Training the tokenizer#

The model never sees text. It sees integers, and the tokenizer is the thing that decides which integers. Byte pair encoding builds that mapping from the corpus itself rather than from a dictionary: start with the 256 possible bytes, count every adjacent pair, merge the most frequent one into a new symbol, and repeat. Each merge adds one entry to the vocabulary. Run it ten thousand times on English and you get whole common words; run it on code and you get four-space indents and double colons.

That is the whole algorithm, and it is short enough to write. The number to watch is bytes per token, because it is the compression ratio that decides how much text a sequence of a given length can hold.

import collections, random

random.seed(0)
WORDS = ("the of and to in a is that it was for on with as by at from but not are "
         "this have has were they their which you one all would there we been when "
         "who will more if out so what about into them then some could time other "
         "than only its over also new because any these two may first well after "
         "council harbour pier storm fishing fleet replacement borrowed destroyed "
         "rebuilt reopened records period debt cleared stone wooden plaque").split()
weights = [1 / (i + 1) for i in range(len(WORDS))]          # a Zipf-ish frequency curve
corpus = " ".join(random.choices(WORDS, weights, k=12000))
held_out = " ".join(random.choices(WORDS, weights, k=2000))

def train_bpe(text, merges):
    seq, rules, nxt = list(text.encode()), [], 256
    for _ in range(merges):
        (a, b), n = collections.Counter(zip(seq, seq[1:])).most_common(1)[0]
        if n < 2:
            break
        out, i = [], 0
        while i < len(seq):
            if i + 1 < len(seq) and seq[i] == a and seq[i + 1] == b:
                out.append(nxt); i += 2
            else:
                out.append(seq[i]); i += 1
        seq, rules, nxt = out, rules + [((a, b), nxt)], nxt + 1
    return rules

def encode(text, rules):
    seq = list(text.encode())
    for (a, b), new in rules:
        out, i = [], 0
        while i < len(seq):
            if i + 1 < len(seq) and seq[i] == a and seq[i + 1] == b:
                out.append(new); i += 2
            else:
                out.append(seq[i]); i += 1
        seq = out
    return seq

def spell(tok, rules):
    table = {new: (a, b) for (a, b), new in rules}
    if tok < 256:
        return chr(tok)
    a, b = table[tok]
    return spell(a, rules) + spell(b, rules)

rules = train_bpe(corpus, 1200)
print("first merges learned:", " ".join(repr(spell(n, rules)) for _, n in rules[:8]))
print()
print("vocab   tokens for 2000 held-out words   bytes/token")
for k in [0, 100, 300, 600, 1200]:
    sub = rules[:k]
    n = len(encode(held_out, sub))
    print(f"{256 + len(sub):5}   {n:31}   {len(held_out.encode())/n:11.2f}")

# first merges learned: ' t' ' th' ' the' ' a' ' o' ' i' ' of' ' w'
#
# vocab   tokens for 2000 held-out words   bytes/token
#   256                              8158          1.00
#   356                              2476          3.29
#   556                              1553          5.25
#   856                              1392          5.86
#  1368                              1313          6.21

The trade behind that curve is worth stating, because it is asked. A larger vocabulary means shorter sequences, which means less compute per document and more text inside a fixed context window. It also means a larger embedding matrix and a larger output projection, and rarer tokens that each get seen fewer times during training. Somewhere in the middle those cross, and the crossing point has moved up as models have grown.

The failure here is specific and permanent, which is what makes it worth a paragraph. The tokenizer is trained once, on a sample of the corpus, and then frozen for the life of the model — you cannot change it later without retraining everything. So the mixture of that sample decides, forever, how efficiently the model can represent anything. A tokenizer fitted mostly to English spends two to four times as many tokens on the same content in a language it saw little of, and every one of those tokens is paid again at training time, at inference time, and against the context window. Measure bytes per token on the kinds of text you actually care about before you freeze it.

The run#

• A tokenizer is trained on a sample of the corpus first, with byte pair encoding.
• The model trains on trillions of tokens with a warmup and a decaying learning rate.
• Teams watch the loss curve for spikes, gradient norms, and small benchmark evaluations every few thousand steps.
• Loss spikes are handled by rewinding to an earlier checkpoint, sometimes skipping the data that caused it.
• Near the end, higher-quality data is often weighted more heavily, a stage sometimes called annealing or midtraining.

What actually goes wrong in a long run#

A pretraining run is a month-long single process, and the interesting failures are not crashes. A crash you notice. The failure that costs a team a week is a loss spike: the curve jumps by orders of magnitude for one step and then either settles back or never recovers. The causes are mundane — a shard that escaped the filters, a batch of near-identical documents, an unlucky interaction between a large gradient and the optimiser's accumulated state — and the reason one bad batch can do permanent damage is that Adam scales updates by the running estimate of gradient magnitude, so a single enormous gradient moves the weights a long way and poisons that estimate for thousands of steps afterwards.

The mechanism is small enough to reproduce on a laptop. Train anything for a hundred steps, hand it one batch whose inputs are five hundred times too large — a shard that arrived unnormalised, which is an ordinary pipeline bug — and watch what the defences do.

import torch, torch.nn as nn

def run(clip=None, skip=False, steps=120, bad_step=60):
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(16, 64), nn.ReLU(), nn.Linear(64, 1))
    opt = torch.optim.Adam(model.parameters(), lr=1e-2)
    w = torch.randn(16, 1)
    history = []
    for step in range(steps):
        g = torch.Generator().manual_seed(step)
        x = torch.randn(32, 16, generator=g)
        if step == bad_step:
            x = x * 500                      # one shard arrived unnormalised
        y = x @ w
        loss = ((model(x) - y) ** 2).mean()
        if skip and history and loss.item() > 20 * sorted(history[-20:])[len(history[-20:]) // 2]:
            history.append(history[-1])      # refuse the batch, keep the weights
            continue
        opt.zero_grad()
        loss.backward()
        if clip:
            nn.utils.clip_grad_norm_(model.parameters(), clip)
        opt.step()
        history.append(loss.item())
    return history

for label, kw in [("no defence", {}), ("clip at 1.0", {"clip": 1.0}),
                  ("clip + skip", {"clip": 1.0, "skip": True})]:
    h = run(**kw)
    print(f"{label:12} step 59 {h[59]:9.3f}   step 60 {h[60]:12.1f}   "
          f"step 70 {h[70]:10.3f}   step 119 {h[119]:9.3f}")

# no defence   step 59     0.136   step 60     147602.5   step 70      5.689   step 119    12.254
# clip at 1.0  step 59     0.113   step 60     180806.3   step 70      0.029   step 119     0.034
# clip + skip  step 59     0.113   step 60          0.1   step 70      0.030   step 119     0.036

Two things in that output are worth separating. Clipping does not stop the loss being reported as 180,806 on the bad batch — the batch really is that wrong — it stops the update from being proportionally enormous, and the run is back under 0.03 within ten steps. Skipping goes further and refuses the batch outright when its loss is far above the recent median, so the spike never appears at all. In a real run you want both, because clipping is a bound on damage and skipping is a bound on how often you pay it.

That gives the standard playbook, which is what an interviewer is after when they ask what you do about a loss spike. Gradient clipping is always on, not switched on after a problem. Checkpoints are frequent, because the recovery is to rewind. When a spike leaves permanent damage you restart from the last good checkpoint and skip forward past the batches that caused it, and if the spike reproduces at the same point in the data, the problem is the data and not the optimiser. Checkpoint frequency is a straight cost calculation: four thousand and ninety-six GPUs at three dollars an hour is about 295,000 dollars a day, so an hour of lost progress is roughly 12,000 dollars and writing a checkpoint every thirty minutes is cheap insurance.

Who should ever do this#

Almost nobody, and being clear about that is more useful than any of the detail above. The reason to learn the pipeline is that it explains model behaviour and it comes up in interviews, not because you are going to run it. The honest ladder looks like this, and the right answer is nearly always as high up it as you can get away with.

approach                 typical cost     changes          right when
prompting                pennies          nothing          the capability is already there
retrieval                a day's work     what it sees     the gap is knowledge, not behaviour
LoRA fine-tune           tens of dollars  behaviour        format, tone, a narrow task
full fine-tune           hundreds         behaviour        LoRA is not enough, rare
continued pretraining    thousands +      knowledge too    a domain genuinely absent from the corpus
pretraining from scratch millions         everything       you need control of every token

There are genuinely three cases for the bottom rung. A domain whose text is not in any public corpus and cannot be — proprietary formats, instrument traces, a language with almost no web presence. A hard requirement to account for the provenance and licence of every training token, which no public base model can give you. And research on the training process itself, where the run is the experiment. Wanting the model to know your documents is not on that list; that is retrieval, and the section on fine-tuning against RAG argues it out.

Continued pretraining is the rung people overlook, and it is often the right one. You take a released base model and keep training it on your own corpus at a low learning rate, usually mixed with general text so it does not forget everything else. It costs thousands rather than millions because you are buying the last few per cent of the token budget rather than all of it, and it is the realistic answer for a domain the base model has seen a little of and not enough.

Run a miniature#

Projects such as nanochat and nanoGPT let you run the whole pipeline at small scale: train a tokenizer, pretrain a small model on a few billion tokens from a public dataset, then fine-tune it to chat. Doing it once turns every paragraph above into something you have seen.

What to keep and what to skip#

• Keep: the order of the stages, and that most of them throw data away rather than add anything.
• Keep: byte pair encoding as repeated merging of the most frequent pair, and that the tokenizer is frozen for the life of the model.
• Keep: clip, checkpoint, rewind and skip, in that order, as the answer to a loss spike.
• Keep: the ladder, and that continued pretraining exists between fine-tuning and starting over.
• Skip: learning-rate schedule folklore. Warmup then cosine decay is the default and the variations are not what you are being asked about.
• Skip: implementing MinHash or a distributed deduplication job unless that is the role you are interviewing for.