Asked about Using scaling laws: the arithmetic
You have a fixed compute budget. How do you choose model size and data size?
An answer that works
Start from compute-optimal scaling, roughly 20 tokens per parameter, which gives the lowest loss for that training budget. Then adjust for inference: if the model will serve heavy traffic, a smaller model trained on more data is cheaper overall. Validate with small-scale runs and a fitted scaling curve before committing.
Then they push further
Estimate the training compute for a 1B parameter model on 100B tokens.
About 6 times 1e9 times 1e11, which is 6e20 FLOPs.
What if you do not have enough unique data?
Repeating data for a few epochs costs relatively little, but returns fall with more repetition. Beyond that, better filtering, synthetic data, or a smaller model are the options.
What gets marked up
Marked up for the 6ND estimate, the compute-optimal ratio, and the inference cost argument for overtraining.
Reading an answer is not being able to give one. The section that teaches this is Using scaling laws: the arithmetic, 4 hours of Scaling laws.
Learn this properly