Asked about The pretraining pipeline, end to end
Walk me through how a pretraining dataset is built.
An answer that works
Start from web crawls, extract main text, identify language, apply heuristic filters, use a quality classifier, deduplicate exactly and approximately, remove personal and unsafe content, then mix in code, books, papers and maths in tuned proportions, and tokenize.
Then they push further
Why does deduplication matter so much?
Duplicates waste compute on repeated content, overweight whatever happens to be copied often, increase memorisation of specific texts, and can leak benchmark items. Removing them improves quality per token.
How would you decide the data mixture?
Train small proxy models on different mixtures and compare their downstream evaluations, then scale up the best. Mixtures are chosen empirically, not by principle.
What gets marked up
Marked up for quality filtering and deduplication as central, and for empirical mixture selection with small models.
Reading an answer is not being able to give one. The section that teaches this is The pretraining pipeline, end to end, 5 hours of Pretraining.
Learn this properlyAlso asked about Pretraining