Training something too big for one GPU

At some point the model, its gradients and its optimiser state stop fitting on one device, and training becomes a distributed systems problem. The mistake that costs the most is reaching for the most sophisticated answer first. There is an order to try things in, each step is more complicated and slower per device than the last, and most people never need to go past the second one. This section gives that order, the three ways of splitting work and what each costs, and the failure modes that only appear once a run lasts days.

Why one device runs out#

Training needs far more memory than inference, and the weights are the smallest part of it. Every parameter carries a gradient, and an optimiser like Adam carries two more numbers per parameter plus a full-precision master copy. Mixed-precision Adam works out at sixteen bytes per parameter, so a seven-billion-parameter model needs about 104 GiB before storing a single activation — more than an 80 GiB card. The arithmetic is worked in full in the numbers section of this topic; the consequence is that the parameter count alone never tells you whether something fits.

what you have                              what to do
fits on one GPU                            one GPU. stop here.
fits, but the batch is too small           gradient accumulation
model fits, optimiser state does not       ZeRO stage 1 or 2 / FSDP
model does not fit                         ZeRO stage 3 / FSDP full shard
one layer does not fit                     tensor parallelism
too many layers for the interconnect       pipeline parallelism

Three ways to split#

The three parallelisms split different things, and the distinction that matters is what each one has to send across the wire and how often.

                  what is split        what crosses the wire     when
data parallel     the batch            gradients, once            per step
tensor parallel   each layer's maths   activations, twice         per layer
pipeline parallel the layers           activations at boundaries  per stage

Data parallelism gives every device a full copy of the model and a different slice of the batch, then averages the gradients. It is the simplest and the only one that does not change the model code. Its limit is memory: every device holds everything, so it cannot help with a model that does not fit. Sharded data parallelism — FSDP, or ZeRO — keeps the same programming model and splits the optimiser state, gradients and optionally the weights across devices, gathering each piece back only while it is needed. That is usually the whole answer, and it is covered in the next section.

Tensor parallelism splits the arithmetic inside a layer: each device holds part of a weight matrix and computes part of the result, and the parts are combined before the layer finishes. It is what you use when a single layer is too large for one device. Because it communicates inside every layer rather than once per step, it is almost always confined to the devices inside one machine, on NVLink. Pipeline parallelism gives each device a contiguous block of layers and passes activations along the chain, which communicates little but introduces the bubble: while device one works on the first micro-batch, the others have nothing to do. Splitting the batch into many micro-batches shrinks the bubble without removing it.

Large training runs combine all three, which is where the phrase 3D parallelism comes from: tensor parallel within a machine, pipeline parallel across a few machines, data parallel across the rest. You do not need to configure that to be hired, but you should be able to say why each sits at the level it does — fastest links for the chattiest parallelism.

Communication is the limit#

The reason the ordering above is not arbitrary is that every step past the first buys memory with bandwidth. Whether that trade is affordable is arithmetic, not judgement: a seven-billion-parameter gradient sync is about 27 ms over NVLink and about 19.6 seconds over 10 Gb Ethernet, which is 6 per cent of a training step against 98 per cent of it. Same model, same eight GPUs, and the interconnect alone decides whether the cluster is useful. Doing that calculation before provisioning is the single highest-value habit in this topic, and the next section but one works it in full.

Long runs fail#

A run measured in days meets failures a run measured in minutes never does, and none of them are exotic: a node dies, a network partition hangs a collective, a disk fills with checkpoints, a GPU develops memory errors. The distributed part makes each one worse, because collectives are synchronous — one slow or dead device stalls all of them, and the symptom is a job that is not crashed and not progressing.

• Checkpoint on time, not on epochs, and keep the optimiser state: a weights-only checkpoint cannot resume Adam correctly.
• Make resuming the normal path and test it on day one, including the data loader position, or a resumed run silently retrains on the same examples.
• Log throughput per step, not just loss. A straggler shows up as throughput decay long before anything fails.
• Set a timeout on collectives. The default in some stacks is half an hour of silence before anything is reported.
• Keep the last few checkpoints rather than only the newest, because a loss spike is usually discovered after the checkpoint that contains it.
• Record the seed, the data order and the exact configuration with the checkpoint. A run you cannot reproduce is a run you cannot debug.

The loss spike deserves its own note, because it is the characteristic failure of a long run and the response is standard: when the loss jumps and does not recover, roll back to the last good checkpoint, skip or reshuffle the batches around the spike, and lower the learning rate for a while. That sequence is what large training reports describe, and being able to say it is what the question is testing.