Asked about Where capability comes from
Roughly what would it cost to pretrain a 7B model?
An answer that works
Compute is about six FLOPs per parameter per token, so seven billion parameters on a compute-optimal 140 billion tokens is about 5.9e21 FLOPs. An H100 does 989 TFLOP/s of bf16 peak and you realise perhaps forty per cent of it, which is roughly 4,100 GPU-hours, so about twelve thousand dollars at three dollars an hour, or under three days on sixty-four cards. A production model trained on ten times the data costs ten times that.
Then they push further
Where does the six come from?
Two FLOPs per parameter per token for the forward pass, since each weight is one multiply and one add, and about twice that again for the backward pass, which computes gradients with respect to both activations and weights.
Why is forty per cent the number you used rather than peak?
Model FLOPs utilisation. Real runs lose throughput to communication, memory movement, pipeline bubbles and kernels that are not at peak. Thirty to fifty per cent is the usual band for a well-tuned large run, and quoting peak FLOPs gives an estimate that is two to three times optimistic.
What gets marked up
Strong answers compute rather than recall, state the utilisation assumption explicitly, and note that the compute-optimal figure is not what a model you can call was actually trained on.
Reading an answer is not being able to give one. The section that teaches this is Where capability comes from, 5 hours of Pretraining.
Learn this properlyAlso asked about Pretraining