FSDP

Also: fully sharded data parallel

PyTorch's implementation of sharded data parallelism: parameters live split across GPUs and are gathered layer by layer for each forward and backward pass, then released.

A definition is not an understanding. This is taught properly in Sharding a model with FSDP, part of Distributed training.

Learn this properly

Terms from the same topic

Gradient accumulationData parallelismTensor parallelismPipeline parallelismZeRONCCL