Asked about Choosing hardware and renting GPUs

You need to serve a 7B model cheaply at moderate traffic. How do you choose hardware?

An answer that works

Check it fits with room for the KV cache at the required context and concurrency, likely 16-bit or quantized weights on a 24 GB or larger card. Then benchmark candidate GPUs with the real serving engine and compare cost per million tokens at the latency target, rather than choosing on hourly price or peak FLOPs.

Reading an answer is not being able to give one. The section that teaches this is Choosing hardware and renting GPUs, 3 hours of GPUs & accelerators.

Learn this properly

Also asked about GPUs & accelerators

Why are GPUs used for deep learning?What is mixed precision training?Your GPU shows 30 percent utilisation. What is likely wrong?Is LLM inference compute-bound or memory-bound?What does kernel fusion do?How do you tell whether a GPU workload is limited by compute or memory bandwidth?