Asked about Data is the model
Why deduplicate a pretraining corpus?
An answer that works
Repeated text is memorised rather than generalised from, and it wastes compute on seeing the same thing many times. Near-duplicates matter as much as exact copies.
Reading an answer is not being able to give one. The section that teaches this is Data is the model, 4 hours of Pretraining.
Learn this properlyAlso asked about Pretraining
How does a language model learn?What are scaling laws?Why do models produce confident wrong answers?Roughly what would it cost to pretrain a 7B model?Given a fixed compute budget, how do you choose model size and token count?What matters most for pretraining quality?Walk me through how a pretraining dataset is built.A new open model claims to beat a larger one on several benchmarks. How do you evaluate the claim?