Tensor parallelism
Also: model parallelism
Splitting individual weight matrices across GPUs so each holds a slice of every layer. It needs a collective communication inside each layer, so it belongs within a fast-interconnect node rather than across nodes.
A definition is not an understanding. This is taught properly in Training something too big for one GPU, part of Distributed training.
Learn this properlyTerms from the same topic