Asked about Sharded data parallelism in practice
What is activation checkpointing?
An answer that works
Storing only some forward-pass activations and recomputing the rest during the backward pass. It trades roughly a third more compute for a large reduction in memory, and is often what makes a model fit without new hardware.
Reading an answer is not being able to give one. The section that teaches this is Sharded data parallelism in practice, 4 hours of Distributed training.
Learn this properlyAlso asked about Distributed training
Why does training need so much more memory than inference?What is the difference between data and tensor parallelism?What usually limits distributed training?What do the ZeRO stages shard?What happens during a DDP training step?When would you use FSDP instead of DDP?Estimate how much memory it takes to train a 7B model with Adam in mixed precision.