Quantization
Storing weights, and sometimes activations, at lower precision — 8-bit or 4-bit instead of 16-bit. Memory drops proportionally, and so does the memory traffic that limits generation speed.
What people get wrong
Quality loss at 8-bit is usually negligible and at 4-bit is small but real, and it hits the long tail hardest. Measure it on your own task rather than on a benchmark average.
A definition is not an understanding. This is taught properly in Memory arithmetic and quantizing a model yourself, part of Quantization & inference optimisation.
Learn this properlyTerms from the same topic