Asked about Training something too big for one GPU
Why does training need so much more memory than inference?
An answer that works
Inference needs the weights. Training also needs gradients, optimiser state which for Adam is several times the model size, and the activations retained for the backward pass.
Reading an answer is not being able to give one. The section that teaches this is Training something too big for one GPU, 5 hours of Distributed training.
Learn this properlyAlso asked about Distributed training
What is the difference between data and tensor parallelism?What usually limits distributed training?What do the ZeRO stages shard?What is activation checkpointing?What happens during a DDP training step?When would you use FSDP instead of DDP?Estimate how much memory it takes to train a 7B model with Adam in mixed precision.