Asked about Where capability comes from
What are scaling laws?
An answer that works
Loss falls predictably as a power law in compute, data and parameters, across many orders of magnitude. They let a lab decide how to spend a compute budget before committing, and the compute-optimal finding showed size and data should grow together.
Reading an answer is not being able to give one. The section that teaches this is Where capability comes from, 5 hours of Pretraining.
Learn this properlyAlso asked about Pretraining
How does a language model learn?Why do models produce confident wrong answers?Roughly what would it cost to pretrain a 7B model?Given a fixed compute budget, how do you choose model size and token count?What matters most for pretraining quality?Why deduplicate a pretraining corpus?Walk me through how a pretraining dataset is built.A new open model claims to beat a larger one on several benchmarks. How do you evaluate the claim?