ZeRO
Also: zero redundancy optimiser
Sharding optimiser state, then gradients, then parameters across data-parallel workers instead of replicating them, gathering each piece only when needed. Cuts memory per GPU by close to the number of workers.
A definition is not an understanding. This is taught properly in Sharded data parallelism in practice, part of Distributed training.
Learn this properlyTerms from the same topic