Asked about Sharded data parallelism in practice
What do the ZeRO stages shard?
An answer that works
Stage 1 shards optimiser state, stage 2 adds gradients, stage 3 adds the weights. Each saves more memory and adds more communication, with weights being the costliest because they must be gathered before every layer.
Reading an answer is not being able to give one. The section that teaches this is Sharded data parallelism in practice, 4 hours of Distributed training.
Learn this properlyAlso asked about Distributed training
Why does training need so much more memory than inference?What is the difference between data and tensor parallelism?What usually limits distributed training?What is activation checkpointing?What happens during a DDP training step?When would you use FSDP instead of DDP?Estimate how much memory it takes to train a 7B model with Adam in mixed precision.