Distributed training interview questions
8 questions across 5 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Training something too big for one GPU
Read the section that teaches thisIf this comes up in an interview
Why does training need so much more memory than inference?
Inference needs the weights. Training also needs gradients, optimiser state which for Adam is several times the model size, and the activations retained for the backward pass.
What is the difference between data and tensor parallelism?
Data parallelism replicates the whole model on each device and splits the batch, requiring the model to fit on one device. Tensor parallelism splits individual layers across devices for models too large for one, at the cost of communication inside every layer.
What usually limits distributed training?
Communication bandwidth. Synchronising gradients every step moves a large amount of data, and across machines that frequently dominates compute time, which is why interconnect drives cluster design.
Sharded data parallelism in practice
Read the section that teaches thisIf this comes up in an interview
What do the ZeRO stages shard?
Stage 1 shards optimiser state, stage 2 adds gradients, stage 3 adds the weights. Each saves more memory and adds more communication, with weights being the costliest because they must be gathered before every layer.
What is activation checkpointing?
Storing only some forward-pass activations and recomputing the rest during the backward pass. It trades roughly a third more compute for a large reduction in memory, and is often what makes a model fit without new hardware.
Data parallel training in PyTorch
Read the section that teaches thisIf this comes up in an interview
What happens during a DDP training step?
Each process runs forward and backward on its own slice of the batch. During the backward pass, gradients are averaged across processes with an all-reduce, overlapped with the remaining backward computation. Every process then applies the same averaged gradient, so the model copies stay identical.
Then they push further
Going from 1 to 8 GPUs only gives 4 times the throughput. Where do you look?
Profile. Common causes are data loading that cannot keep up, gradient communication that is not overlapped or crosses a slow link, a small model or batch per GPU so communication dominates, and synchronisation points such as logging that stall every step.
What changes about the learning rate?
The global batch is now eight times larger. Many setups increase the learning rate with batch size, often with warmup, but it has to be retuned rather than assumed.
What gets marked up. Marked up for all-reduce during backward, the batch size consequence, and a method for finding the bottleneck.
Sharding a model with FSDP
Read the section that teaches thisIf this comes up in an interview
When would you use FSDP instead of DDP?
When the model's weights, gradients and optimiser state do not fit on one GPU with plain replication. FSDP shards all three and gathers each layer's weights just in time, at the cost of extra communication.
Then they push further
Your FSDP job fits in memory but is much slower than expected across two machines. Why?
Parameter gathers and gradient reduce-scatters in every layer cross the slower network link. Shard within each machine and replicate across machines, increase the work per GPU, and check that communication overlaps with compute.
What gets marked up. Marked up for knowing what is sharded, the per-layer gather cost, and hybrid sharding as the fix across nodes.
Collectives and the numbers that decide a design
Read the section that teaches thisIf this comes up in an interview
Estimate how much memory it takes to train a 7B model with Adam in mixed precision.
About 16 bytes per parameter: 2 for 16-bit weights, 2 for gradients, 4 for 32-bit master weights and 8 for Adam's two moments. That is about 112 GB before activations, so it needs sharding across several GPUs.
Then they push further
How long would a gradient all-reduce take across 8 GPUs over a 100 Gbit/s network?
Each GPU moves about 2 times 7/8 of 14 GB, around 24 GB. At about 12.5 GB/s that is roughly two seconds per step, which is why this is done over fast in-machine links or overlapped with computation.
What is MFU and what would a low value tell you?
The share of peak hardware FLOPs spent on model computation. A low value means time goes to communication, data loading, idle pipeline stages or inefficient kernels, and says where to look.
What gets marked up. Strong candidates do the arithmetic out loud with the 16 bytes per parameter figure and relate communication time to interconnect bandwidth.
These are the questions for one topic. The whole question bank covers every topic on the site.