Asked about Getting the most from the hardware
Is LLM inference compute-bound or memory-bound?
An answer that works
Generating one token at a time is typically memory-bound, since each step reads all the weights to do comparatively little arithmetic. Batching many requests together is the main way to make it efficient.
Reading an answer is not being able to give one. The section that teaches this is Getting the most from the hardware, 4 hours of GPUs & accelerators.
Learn this properlyAlso asked about GPUs & accelerators
Why are GPUs used for deep learning?What is mixed precision training?Your GPU shows 30 percent utilisation. What is likely wrong?What does kernel fusion do?How do you tell whether a GPU workload is limited by compute or memory bandwidth?You need to serve a 7B model cheaply at moderate traffic. How do you choose hardware?