Mixed precision
Training with most arithmetic in 16-bit while keeping a master copy of the weights and the accumulations in 32-bit. Roughly halves memory and speeds up the maths without changing the result much.
A definition is not an understanding. This is taught properly in What the hardware is doing, part of GPUs & accelerators.
Learn this properlyTerms from the same topic