Data parallelism
Putting a full copy of the model on every GPU, giving each a different slice of the batch, and averaging the gradients. The simplest way to scale, and the right one until the model stops fitting on one device.
A definition is not an understanding. This is taught properly in Data parallel training in PyTorch, part of Distributed training.
Learn this properlyTerms from the same topic