GPUs & accelerators interview questions
7 questions across 4 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
What the hardware is doing
Read the section that teaches thisIf this comes up in an interview
Why are GPUs used for deep learning?
The core operation is matrix multiplication, which is thousands of independent multiply-accumulate operations. A GPU has thousands of simple cores suited to exactly that, while a CPU has a few complex ones.
What is mixed precision training?
Using 16-bit for most operations and 32-bit where precision matters, with loss scaling to stop small gradients underflowing. It roughly halves memory and can double throughput at essentially no quality cost.
Your GPU shows 30 percent utilisation. What is likely wrong?
The GPU is waiting, almost always for data. Raise data loader workers, enable pinned memory, and check for per-item preprocessing that should be precomputed or moved onto the device.
Getting the most from the hardware
Read the section that teaches thisIf this comes up in an interview
Is LLM inference compute-bound or memory-bound?
Generating one token at a time is typically memory-bound, since each step reads all the weights to do comparatively little arithmetic. Batching many requests together is the main way to make it efficient.
What does kernel fusion do?
Combines several operations into one kernel so intermediate results stay on-chip instead of being written to and read from memory between steps. The saving is in memory traffic rather than arithmetic.
Measuring a GPU workload
Read the section that teaches thisIf this comes up in an interview
How do you tell whether a GPU workload is limited by compute or memory bandwidth?
Estimate its arithmetic intensity, FLOPs per byte moved, and compare with the hardware's ratio of peak compute to bandwidth. Then confirm with a profiler: memory-bound kernels show high bandwidth use and low compute utilisation.
Then they push further
Why is LLM token generation memory-bound?
Each step processes one new token per sequence but must read every weight from memory, so the FLOPs per byte are very low. Batching many sequences reuses each weight read for more work, raising intensity.
What gets marked up. Marked up for the FLOPs per byte reasoning and connecting it to batching.
Choosing hardware and renting GPUs
Read the section that teaches thisIf this comes up in an interview
You need to serve a 7B model cheaply at moderate traffic. How do you choose hardware?
Check it fits with room for the KV cache at the required context and concurrency, likely 16-bit or quantized weights on a 24 GB or larger card. Then benchmark candidate GPUs with the real serving engine and compare cost per million tokens at the latency target, rather than choosing on hourly price or peak FLOPs.
These are the questions for one topic. The whole question bank covers every topic on the site.