Memory-bound

A workload limited by how fast data can be moved from memory rather than by arithmetic. Single-stream token generation is memory-bound, which is why a faster chip does not help it much but quantization does.

A definition is not an understanding. This is taught properly in Getting the most from the hardware, part of GPUs & accelerators.

Learn this properly

Terms from the same topic

Mixed precision GPU VRAM HBM FLOPs Compute-bound Roofline model