Asked about Training something too big for one GPU
What is the difference between data and tensor parallelism?
An answer that works
Data parallelism replicates the whole model on each device and splits the batch, requiring the model to fit on one device. Tensor parallelism splits individual layers across devices for models too large for one, at the cost of communication inside every layer.
Reading an answer is not being able to give one. The section that teaches this is Training something too big for one GPU, 5 hours of Distributed training.
Learn this properlyAlso asked about Distributed training
Why does training need so much more memory than inference?What usually limits distributed training?What do the ZeRO stages shard?What is activation checkpointing?What happens during a DDP training step?When would you use FSDP instead of DDP?Estimate how much memory it takes to train a 7B model with Adam in mixed precision.