FLOPs

Also: floating point operations

A count of arithmetic operations, used as the unit of compute for training and inference. A forward pass costs roughly two FLOPs per parameter per token, and training about three times that again.

What people get wrong

Careful with the plural: FLOPs is a quantity of work, FLOPS is a rate per second. A chip's advertised FLOPS figure also assumes ideal conditions you will not reach.

A definition is not an understanding. This is taught properly in What the hardware is doing, part of GPUs & accelerators.

Learn this properly

Terms from the same topic

Mixed precision GPU VRAM HBM Memory-bound Compute-bound Roofline model