Asked about Training something too big for one GPU
What usually limits distributed training?
An answer that works
Communication bandwidth. Synchronising gradients every step moves a large amount of data, and across machines that frequently dominates compute time, which is why interconnect drives cluster design.
Reading an answer is not being able to give one. The section that teaches this is Training something too big for one GPU, 5 hours of Distributed training.
Learn this properlyAlso asked about Distributed training
Why does training need so much more memory than inference?What is the difference between data and tensor parallelism?What do the ZeRO stages shard?What is activation checkpointing?What happens during a DDP training step?When would you use FSDP instead of DDP?Estimate how much memory it takes to train a 7B model with Adam in mixed precision.