Asked about Sharding a model with FSDP
When would you use FSDP instead of DDP?
An answer that works
When the model's weights, gradients and optimiser state do not fit on one GPU with plain replication. FSDP shards all three and gathers each layer's weights just in time, at the cost of extra communication.
Then they push further
Your FSDP job fits in memory but is much slower than expected across two machines. Why?
Parameter gathers and gradient reduce-scatters in every layer cross the slower network link. Shard within each machine and replicate across machines, increase the work per GPU, and check that communication overlaps with compute.
What gets marked up
Marked up for knowing what is sharded, the per-layer gather cost, and hybrid sharding as the fix across nodes.
Reading an answer is not being able to give one. The section that teaches this is Sharding a model with FSDP, 4 hours of Distributed training.
Learn this properlyAlso asked about Distributed training