Asked about Measuring a GPU workload

How do you tell whether a GPU workload is limited by compute or memory bandwidth?

An answer that works

Estimate its arithmetic intensity, FLOPs per byte moved, and compare with the hardware's ratio of peak compute to bandwidth. Then confirm with a profiler: memory-bound kernels show high bandwidth use and low compute utilisation.

Then they push further

Why is LLM token generation memory-bound?

Each step processes one new token per sequence but must read every weight from memory, so the FLOPs per byte are very low. Batching many sequences reuses each weight read for more work, raising intensity.

What gets marked up

Marked up for the FLOPs per byte reasoning and connecting it to batching.

Reading an answer is not being able to give one. The section that teaches this is Measuring a GPU workload, 4 hours of GPUs & accelerators.

Learn this properly

Also asked about GPUs & accelerators

Why are GPUs used for deep learning?What is mixed precision training?Your GPU shows 30 percent utilisation. What is likely wrong?Is LLM inference compute-bound or memory-bound?What does kernel fusion do?You need to serve a 7B model cheaply at moderate traffic. How do you choose hardware?