Asked about Data parallel training in PyTorch
What happens during a DDP training step?
An answer that works
Each process runs forward and backward on its own slice of the batch. During the backward pass, gradients are averaged across processes with an all-reduce, overlapped with the remaining backward computation. Every process then applies the same averaged gradient, so the model copies stay identical.
Then they push further
Going from 1 to 8 GPUs only gives 4 times the throughput. Where do you look?
Profile. Common causes are data loading that cannot keep up, gradient communication that is not overlapped or crosses a slow link, a small model or batch per GPU so communication dominates, and synchronisation points such as logging that stall every step.
What changes about the learning rate?
The global batch is now eight times larger. Many setups increase the learning rate with batch size, often with warmup, but it has to be retuned rather than assumed.
What gets marked up
Marked up for all-reduce during backward, the batch size consequence, and a method for finding the bottleneck.
Reading an answer is not being able to give one. The section that teaches this is Data parallel training in PyTorch, 5 hours of Distributed training.
Learn this properlyAlso asked about Distributed training