Asked about Where capability comes from
Given a fixed compute budget, how do you choose model size and token count?
An answer that works
C equals 6ND fixes the trade, so choosing a size chooses the token count. The compute-optimal point is around twenty tokens per parameter, which for a 1e22 FLOP budget lands near nine billion parameters. Then adjust for serving: if the model will be called heavily, go smaller and train longer, because training is paid once and inference is paid every request.
Reading an answer is not being able to give one. The section that teaches this is Where capability comes from, 5 hours of Pretraining.
Learn this properlyAlso asked about Pretraining
How does a language model learn?What are scaling laws?Why do models produce confident wrong answers?Roughly what would it cost to pretrain a 7B model?What matters most for pretraining quality?Why deduplicate a pretraining corpus?Walk me through how a pretraining dataset is built.A new open model claims to beat a larger one on several benchmarks. How do you evaluate the claim?