Choosing hardware and renting GPUs

Choosing a GPU is four questions in a fixed order, and only the last one is about price. Does the job fit in the memory; is the work limited by bandwidth or by arithmetic; do the cards need to talk to each other; and only then, what does a unit of useful work cost. Taken in that order the decision is usually forced. Taken in the other order — starting from the hourly rate — you rent the cheapest card, discover the model does not fit, and pay for the week it takes to find out.

Memory first#

Memory is a hard constraint rather than a preference: below the threshold the job does not run at all, and above it the extra is worth nothing. So work out the requirement before looking at any other column. For inference the rule is roughly two bytes per parameter in bf16, plus the KV cache, which grows with batch and context. For training it is about sixteen bytes per parameter with mixed-precision Adam, plus activations — which is why a 7B model needs 104 GiB and will not train on a single 80 GiB card however cheap it is.

job                                   memory needed        smallest sensible card
7B inference, bf16, short context     ~15 GiB              24 GB (L4, A10G)
7B inference, long context, batched   ~30 GiB              40-80 GB
7B LoRA fine-tune                     ~25 GiB              40 GB
7B full fine-tune, Adam               ~104 GiB             multiple 80 GB
70B inference, 4-bit                  ~40 GiB              48-80 GB

Bandwidth for inference#

Generating one token reads every weight the model has and does a small amount of arithmetic with each. That makes single-stream generation memory-bound: the ceiling is how fast the weights can be moved out of memory, not how fast the chip can multiply. A 7B model in bf16 is about 14 GB, so a card with 2 TB/s can in principle read it around 140 times a second, and the achievable token rate follows from that number rather than from the TFLOPs.

Two consequences follow, and both are counterintuitive until the arithmetic is in front of you. First, for serving a single stream a card with more bandwidth beats a card with more compute, even at the same price. Second, batching is close to free on the compute side: a batch of sixteen reads the same weights once and does sixteen times the arithmetic, so throughput rises nearly linearly until the arithmetic finally becomes the limit. That is why inference services batch aggressively and why the per-request cost falls so sharply with load.

Interconnect for multi-GPU training#

The moment a job spans more than one card, what connects them matters more than the cards. The numbers section of the distributed training topic works this out: a 7B gradient sync is about 27 ms over NVLink and about 19.6 seconds over 10 Gb Ethernet — 6 per cent of a step against 98 per cent of it. Eight cards on a slow network is not a cluster, it is eight cards waiting.

Cost per useful unit#

Now, and only now, price. The rate per hour is not the cost of the work; the cost of the work is the rate divided by how much work the card does in that hour. Here is the same training job — ten exaFLOPs, a realistic forty per cent utilisation — priced across a range of cards.

# Approximate on-demand rates. These move constantly; the arithmetic does not.
gpus = [  # name,     VRAM, bf16 TFLOP/s, GB/s, $/hour
    ("T4",         16,   65,  300, 0.35),
    ("L4",         24,  121,  300, 0.80),
    ("A10G",       24,  125,  600, 1.00),
    ("A100 40GB",  40,  312, 1555, 2.50),
    ("A100 80GB",  80,  312, 2039, 3.50),
    ("H100 80GB",  80,  989, 3350, 6.00),
]

for name, vram, tflops, bw, price in gpus:
    hours = 1e19 / (tflops * 1e12 * 0.4) / 3600      # 1e19 FLOPs at 40% MFU
    fits = "" if vram >= 80 else "   <- 7B + Adam does not fit"
    cost = hours * price
    print(f"{name:11} {hours:6.1f} h   " "$" f"{cost:7.2f}{fits}")

# T4          106.8 h   $  37.39   <- 7B + Adam does not fit
# L4           57.4 h   $  45.91   <- 7B + Adam does not fit
# A10G         55.6 h   $  55.56   <- 7B + Adam does not fit
# A100 40GB    22.3 h   $  55.64   <- 7B + Adam does not fit
# A100 80GB    22.3 h   $  77.90
# H100 80GB     7.0 h   $  42.13

The T4 row is the trap in the other direction. It shows the lowest total, and it cannot hold the model, so that figure is for a job it will never run. Apply the memory filter first and then compare cost, or you will reason carefully about a column that does not apply. The same table also shows why the A100 80GB is often the worst purchase on the list for compute-heavy work: it is priced for its memory and has the same arithmetic throughput as the 40GB card.

Spot instances and idle GPUs#

Two things dominate a real GPU bill and neither is the hourly rate. The first is spot capacity: interruptible instances are typically sixty to ninety per cent cheaper, and they are entirely usable for training that checkpoints properly — which is the same discipline a long run needs anyway. If your job cannot survive being killed, fixing that is worth more than any negotiation on price.

The second is idle time, which is where most money actually goes. A development instance left running overnight bills identically to one doing useful work, and a notebook attached to an A100 for a week of intermittent typing is the single most common line item on a surprising bill. The defences are unglamorous and effective.

• Put an idle shutdown on every interactive instance. Most platforms have one; it is off by default.
• Develop on the smallest card that runs the code, and move to the large one only for the real run.
• Tag everything by owner and job, and look at the bill weekly. Untagged spend is nobody's and grows.
• Prefer a job that exits to a session you leave open. A queued batch job cannot be forgotten.
• Check utilisation before renting more: a card at 30 per cent is a data-loading problem, and a second card will also sit at 30 per cent.

That last one is the whole topic in a sentence. More hardware multiplies whatever efficiency you already have, so a job running at thirty per cent utilisation on one card runs at thirty per cent on eight, for eight times the money. Measure first.