Pipeline parallelism
Giving each GPU a consecutive group of layers and passing activations along the chain. Communication is small, but devices sit idle waiting for work unless the batch is split into micro-batches.
A definition is not an understanding. This is taught properly in Training something too big for one GPU, part of Distributed training.
Learn this properlyTerms from the same topic