Asked about Collectives and the numbers that decide a design
Estimate how much memory it takes to train a 7B model with Adam in mixed precision.
An answer that works
About 16 bytes per parameter: 2 for 16-bit weights, 2 for gradients, 4 for 32-bit master weights and 8 for Adam's two moments. That is about 112 GB before activations, so it needs sharding across several GPUs.
Then they push further
How long would a gradient all-reduce take across 8 GPUs over a 100 Gbit/s network?
Each GPU moves about 2 times 7/8 of 14 GB, around 24 GB. At about 12.5 GB/s that is roughly two seconds per step, which is why this is done over fast in-machine links or overlapped with computation.
What is MFU and what would a low value tell you?
The share of peak hardware FLOPs spent on model computation. A low value means time goes to communication, data loading, idle pipeline stages or inefficient kernels, and says where to look.
What gets marked up
Strong candidates do the arithmetic out loud with the 16 bytes per parameter figure and relate communication time to interconnect bandwidth.
Reading an answer is not being able to give one. The section that teaches this is Collectives and the numbers that decide a design, 4 hours of Distributed training.
Learn this properlyAlso asked about Distributed training