Getting the most from the hardware

You have a rough model of the hardware; this is about getting work out of it. The first question for any slow workload is whether the processors are the limit or moving data is, because the fixes point in opposite directions. The large matrix multiplications in training are usually compute-bound. Generating one token at a time is memory-bound, which is why batching requests together is the single largest lever on inference cost there is. This section covers making that diagnosis and what follows from each answer.

Two kinds of bottleneck#

A workload is compute-bound when the processors are the limit, and memory-bound when moving data is. Large matrix multiplications in training are usually compute-bound. Generating one token at a time in inference is usually memory-bound, since each step reads all the weights to do relatively little arithmetic.

Training, large batches     compute-bound   more FLOPs help
Inference, one token        memory-bound    more bandwidth helps
                                            batching helps most

You do not have to guess which one you have. Count the arithmetic and count the bytes, divide, and compare with the hardware's own ratio of peak arithmetic to peak bandwidth. Everything below the hardware's number is memory-bound.

operation                              FLOPs        bytes moved    intensity

add two n x n bf16 tensors             n^2          6 n^2          0.17
multiply two n x n bf16 matrices       2 n^3        6 n^2          n / 3
  n = 512                                                            171
  n = 4096                                                         1,365

A GPU with roughly 300 TFLOP/s and 2 TB/s balances at about 150 FLOP/byte.
Below that line, more arithmetic units buy you nothing at all.

The size of the gap is easy to underestimate until you measure it. There is no GPU here, so this is the same experiment on this machine's CPU, where the ratio is different but the physics is identical.

matmul  1024x1024    1,194 GFLOP/s     intensity 171 FLOP/byte
matmul  2048x2048      917 GFLOP/s     intensity 341 FLOP/byte
matmul  4096x4096      769 GFLOP/s     intensity 683 FLOP/byte
add     2048x2048        4.2 GFLOP/s   intensity 0.08          50.3 GB/s
add     4096x4096        4.0 GFLOP/s   intensity 0.08          48.3 GB/s
add     8192x8192        4.0 GFLOP/s   intensity 0.08          48.3 GB/s

Applying this to a language model gives the result people find counterintuitive. Generating one token for one sequence must read every weight in the model — 14 GB for a 7B model in bf16 — and does about two arithmetic operations per weight. Two FLOPs per two bytes is an intensity of 1, two orders of magnitude below the hardware's balance point, so the GPU's arithmetic units are idle more than 99 per cent of the time and generation speed is decided purely by how fast weights can be read.

Batch size fills the machine#

Small batches leave most of the GPU idle. Larger batches keep the cores busy, up to the limit of memory. For inference, batching many requests together is what turns a memory-bound workload into an efficient one.

The mechanism is worth spelling out because it explains why the effect is so large. The weights are read once per step regardless of how many sequences are in the batch. One sequence does one token's worth of arithmetic per weight read; thirty-two sequences do thirty-two tokens' worth for the same read. The arithmetic intensity rises in direct proportion to the batch size, and until it crosses the hardware's balance point the step takes very nearly the same wall-clock time.

batch   weight reads   tokens produced   intensity    time per step   tokens/s
   1      14 GB              1              1x            same           1x
   4      14 GB              4              4x            same           4x
  16      14 GB             16             16x            about same    ~16x
  64      14 GB             64             64x            rising         <64x
 256      14 GB            256            256x            compute-bound  flattens

Two costs push back. Latency for an individual request goes up, because a request waits for the batch to form and then shares the step with everyone else, so throughput and time-to-first-token trade against each other directly. And memory goes up: each sequence in the batch carries its own KV cache, which for a 7B model at 8,192 tokens of context is about 4 GiB per sequence with full multi-head attention. Sixteen concurrent sequences at that context need 64 GiB of cache against 12.6 GiB of weights, so the cache, not the model, is what fills the card.

Fusion#

Each separate operation reads its input from memory and writes its output back. Fusing several operations into one kernel keeps intermediate results on the chip, avoiding round trips. This is much of what compilers and optimised attention implementations do.

Count the traffic on a sequence you write every day. A linear layer followed by a bias add, a GELU and a dropout is four operations on the same tensor. Unfused, the intermediate result is written to memory and read back three times — six tensor-sized transfers that exist only because the operations were separate kernels. Fused, it is written once. The arithmetic is identical; roughly five-sixths of the memory traffic is gone, and since the sequence was memory-bound, that is very close to a five-fold speedup on that part of the network.

In practice you get this by asking rather than by writing kernels. torch.compile fuses elementwise chains automatically, and the attention implementations shipped with modern frameworks are already fused. The engineering skill is recognising which parts of your model are a chain of small elementwise operations, because those are the parts where a compiler has something to work with, and knowing that it will do nothing for a model that is already one large matrix multiplication.

When one GPU is not enough#

Before reaching for more hardware, exhaust the cheap moves, because they are usually enough and they are always faster to try. Mixed precision halves the activation memory and roughly doubles matrix multiply throughput. Gradient accumulation gets you a large effective batch with small resident activations. Activation checkpointing trades about a third more compute for a large memory saving. Each is one line and none of them requires a second device.

When you do add devices, the first thing to know is that the link between them is a hierarchy and it decides what is worth doing. Cards inside one machine talk over a fast direct interconnect at hundreds of gigabytes a second. Machines talk over a network, which is one or two orders of magnitude slower. A parallelism strategy that is fine inside a box can be unusable across boxes, and that boundary, not the number of GPUs, is the thing to design around.

What to skip: writing kernels. The skill that pays here is diagnosis — deciding which side of the roofline you are on, and choosing between a larger batch, a fused implementation, lower precision or more devices. Reading a kernel occasionally is useful; writing one is a specialisation very few roles ask for.