Gradient accumulation
Summing gradients over several small batches before stepping the optimiser, to get the behaviour of a large batch on a GPU that cannot hold one.
A definition is not an understanding. This is taught properly in Data parallel training in PyTorch, part of Distributed training.
Learn this properlyTerms from the same topic