GPUs & accelerators

Everything above this layer runs on a GPU, and a rough model of what one is doing answers the two questions you will be asked constantly: why is this slow, and will it fit. This section covers why matrix maths suits thousands of simple cores, why memory and the bandwidth to it usually bind before arithmetic does, what mixed precision buys and what it costs, and how to read nvidia-smi without being misled — high utilisation can mean the device is busy doing very little.

Many small operations at once#

A CPU has a few very capable cores. A GPU has thousands of simple ones. Matrix multiplication is thousands of independent multiply-and-add operations, which is exactly the shape a GPU is built for.

It is worth being specific about what 'capable' and 'simple' mean, because the difference is not that GPU cores are slower. A CPU core spends most of its transistors on making a single sequence of instructions finish sooner: predicting which branch will be taken, reordering instructions, and holding several megabytes of cache per core so it rarely waits for memory. All of that is machinery for code where the next step depends on the last one. A GPU spends those transistors on more arithmetic units instead, and handles waiting by keeping thousands of threads in flight so there is always one ready to run while others wait.

That bargain only pays when the work is thousands of identical independent operations with no branching. Multiplying two matrices is exactly that — every output element is an independent dot product of a row and a column — which is why the same hardware serves graphics and neural networks. Give a GPU a loop with a data-dependent branch in it and it will lose to a CPU comfortably.

Two consequences follow immediately and explain a lot of otherwise mysterious behaviour. Anything you write as a Python loop over items is the wrong shape and will be slow no matter what device it runs on, because it is a sequence of tiny operations rather than one large one. And a small operation on a GPU costs roughly the same as a medium one, because the fixed cost of launching the work dominates — which is why a batch of 1 and a batch of 8 often take almost the same time.

Memory is usually the constraint#

• VRAM decides whether a model fits at all. Exceeding it fails immediately rather than degrading.
• Bandwidth decides speed for large models, because generation is dominated by reading weights.
• Moving data between host and device is slow. Do it once per batch, never per item.

The second point is the one people find surprising, and it has a simple form. Every operation has an arithmetic intensity: the number of floating-point operations it performs divided by the number of bytes it has to move to and from memory. A chip has a ratio too — its peak arithmetic rate divided by its memory bandwidth. If your operation's intensity is below the chip's ratio, you are memory-bound and buying more arithmetic buys you nothing.

The gap between the two regimes is much larger than intuition suggests. There is no GPU on the machine these words were written on, so here it is measured on its CPU, where the same physics applies with smaller numbers.

matmul  1024x1024     1,194 GFLOP/s      intensity  171 FLOP/byte
matmul  4096x4096       769 GFLOP/s      intensity  683 FLOP/byte
add     4096x4096         4.0 GFLOP/s    intensity 0.08 FLOP/byte    48 GB/s
add     8192x8192         4.0 GFLOP/s    intensity 0.08 FLOP/byte    48 GB/s

This single ratio explains most of what you will be asked about performance. Training with a large batch does big matrix multiplications and is compute-bound. Generating one token at a time reads every weight in the model to do a handful of arithmetic per weight, so it is heavily memory-bound, which is why token generation speed tracks memory bandwidth rather than FLOPs, and why batching many requests together is the largest single lever on inference cost. And it is why fusing operations helps: the arithmetic is unchanged, the round trips to memory are what disappeared.

What is actually in the memory during training#

Ask someone how much memory it takes to train a model and they will usually answer with the parameter count times a couple of bytes. That answer is wrong by roughly a factor of eight, and knowing why is the difference between planning a run and being surprised by it. There are four things in memory, and the weights are the smallest of them.

import torch
import torch.nn as nn

model = nn.Sequential(nn.Linear(1024, 4096), nn.GELU(), nn.Linear(4096, 1024))
opt = torch.optim.AdamW(model.parameters(), lr=1e-4)
model(torch.randn(8, 1024)).pow(2).mean().backward()
opt.step()                                   # the state exists only after a step

n = sum(p.numel() for p in model.parameters())
params = sum(p.numel() * p.element_size() for p in model.parameters())
grads = sum(p.grad.numel() * p.grad.element_size() for p in model.parameters())
state = sum(v.numel() * v.element_size()
            for s in opt.state.values() for v in s.values() if torch.is_tensor(v))
for label, b in [("parameters", params), ("gradients", grads), ("Adam state", state)]:
    print(f"{label:12} {b / 2**20:8.1f} MiB   {b / n:5.1f} bytes per parameter")

# parameters       32.0 MiB     4.0 bytes per parameter
# gradients        32.0 MiB     4.0 bytes per parameter
# Adam state       64.0 MiB     8.0 bytes per parameter
#                                16.0 in total, and the model is 4 of them

In mixed precision the bookkeeping changes but the total does not, which catches people out. Weights and gradients drop to 2 bytes each, but a 32-bit master copy of the weights has to be kept for the optimiser to update accurately, and Adam's two moments stay in 32-bit. Two plus two plus four plus four plus four is still sixteen bytes per parameter. Mixed precision buys speed and cheaper activations; it does not shrink the optimiser.

A 6.74B-parameter model (llama-7B shape), mixed precision, AdamW:

  bf16 weights          2 B/param     12.6 GiB
  bf16 gradients        2 B/param     12.6 GiB
  fp32 master weights   4 B/param     25.1 GiB
  Adam m                4 B/param     25.1 GiB
  Adam v                4 B/param     25.1 GiB
                       --------------------------
  total                16 B/param    100.4 GiB     before a single activation

  optimiser's share: 75 per cent.
  inference on the same model: 12.6 GiB.

The fourth item is activations: everything the forward pass computed that the backward pass needs in order to differentiate. Unlike the other three it does not scale with the parameter count at all — it scales with batch size and sequence length, which means it is the one you control.

Measured by summing every layer output in a 162M transformer, fp32:

  batch 1, 256 tokens      150.3 MiB
  batch 2, 256 tokens      299.9 MiB
  batch 4, 256 tokens      599.1 MiB
  batch 8, 256 tokens    1,197.4 MiB      -> 149.7 MiB per sample, flat

So the batch size you can use is decided by what is left after the fixed 16 bytes per parameter, and the three ways out are all forms of the same trade. Activation checkpointing stores only some activations and recomputes the rest in the backward pass, buying a large memory saving for roughly a third more compute. Gradient accumulation runs several small batches and steps once, so the effective batch is large while the resident activations are small. Sharding the optimiser state across devices attacks the other 12 bytes. Reach for them in that order.

Mixed precision#

Using 16-bit numbers for most operations and 32-bit where precision matters roughly halves memory and can double throughput, with essentially no quality loss. It is close to free and frequently left switched off.

There are two 16-bit formats and they are not interchangeable. Both use 16 bits; they spend them differently, one on range and the other on precision, and which one you have decides whether you need loss scaling.

import torch

for dt in (torch.float32, torch.float16, torch.bfloat16):
    i = torch.finfo(dt)
    print(f"{str(dt):16} max {i.max:.2e}  smallest normal {i.tiny:.2e}  eps {i.eps:.2e}")

# torch.float32    max 3.40e+38  smallest normal 1.18e-38  eps 1.19e-07
# torch.float16    max 6.55e+04  smallest normal 6.10e-05  eps 9.77e-04
# torch.bfloat16   max 3.39e+38  smallest normal 1.18e-38  eps 7.81e-03

g = torch.tensor([2.0e-8, 1.0e-5, 3.0e-3])          # plausible gradient magnitudes
show = lambda t: [f"{v:.2e}" for v in t.tolist()]
print(show(g.half().float()))
print(show(g.bfloat16().float()))
print(show((g * 1024).half().float() / 1024))       # the same in fp16, loss-scaled

# ['0.00e+00', '1.00e-05', '3.00e-03']   fp16: the first gradient became zero
# ['2.00e-08', '1.00e-05', '3.01e-03']   bf16: kept, with range to spare
# ['2.00e-08', '1.00e-05', '3.00e-03']   fp16 scaled by 1024 first: kept

print(float(torch.full((65536,), 1.0, dtype=torch.float16).sum(dtype=torch.float16)))
# inf  -- fp16 cannot represent 65536; bfloat16 and float32 both return 65536.0

The rule follows directly. On hardware with bfloat16 support, use it and do not use a gradient scaler — there is nothing to scale for. On older hardware you have float16, and you need the scaler: multiply the loss by a large constant so small gradients land inside the representable range, then divide it back out before the optimiser step. PyTorch's GradScaler does that and also skips any step whose gradients came back as infinity, adjusting the constant as it goes.

# Needs a CUDA device. model, opt, loss_fn, batch and target are yours.

# On modern hardware, with bfloat16: no scaler needed.
with torch.autocast("cuda", dtype=torch.bfloat16):
    loss = loss_fn(model(batch), target)
loss.backward()
opt.step()
opt.zero_grad()

# On older hardware, with float16: the scaler is required.
scaler = torch.amp.GradScaler()

with torch.autocast("cuda", dtype=torch.float16):
    loss = loss_fn(model(batch), target)

scaler.scale(loss).backward()      # multiply the loss so gradients stay in range
scaler.step(opt)                   # unscale, then skip the step if any gradient is inf
scaler.update()                    # adjust the scale for next time
opt.zero_grad()

Utilisation is not the whole story#

nvidia-smi                       # memory used, utilisation, temperature
watch -n 1 nvidia-smi            # live
nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu,power.draw \
           --format=csv -l 1     # just the four numbers, once a second

Four numbers are worth reading, and the useful skill is knowing what each one rules out rather than what it proves. Memory used against memory total tells you how much headroom the batch size has. Utilisation tells you what fraction of the last sampling interval had any kernel running. Power draw is the honest proxy for whether the arithmetic units are busy, because a GPU doing real matrix multiplication draws close to its rated power and one shuffling memory around does not. Temperature matters only when the card is throttling.

utilisation   power      what it means                      where to look

  low         low      the GPU is waiting                 data loading, or Python
                       for work                           between kernel launches
  high        low      kernels run constantly but         many tiny operations,
                       do little arithmetic               or a memory-bound workload
  high        high     the hardware is genuinely busy     the model, the batch size,
                                                          or accept it and scale out
  any        at cap    thermal or power throttling        cooling, or the power limit

So the diagnostic order is: check memory first, since it decides what is possible at all; then check utilisation, since a low number is unambiguous and almost always means data loading; and only then reach for a profiler, because a high utilisation number tells you nothing on its own.

How deep to go#

For engineering, this level is sufficient: what fits, what makes it slow, and how to see it. Writing CUDA kernels is a specialisation that very few roles require, and it is a poor place to start.

What is worth memorising is short: sixteen bytes per parameter to train with Adam, two bytes to serve; activations scale with batch and sequence and nothing else; a chip balances at roughly a hundred and fifty FLOPs per byte, so anything below that is memory-bound; and bfloat16 has range while float16 has precision. What can be looked up on the day: the datasheet numbers for any particular card, the CUDA and driver version matrix, the flag names for any given framework's mixed precision, and everything about writing kernels.

You have a rough model of the hardware; this is about getting work out of it. The first question for any slow workload is whether the processors are the limit or moving data is, because the fixes point in opposite directions. The large matrix multiplications in training are usually compute-bound. Generating one token at a time is memory-bound, which is why batching requests together is the single largest lever on inference cost there is. This section covers making that diagnosis and what follows from each answer.

Two kinds of bottleneck#

A workload is compute-bound when the processors are the limit, and memory-bound when moving data is. Large matrix multiplications in training are usually compute-bound. Generating one token at a time in inference is usually memory-bound, since each step reads all the weights to do relatively little arithmetic.

Training, large batches     compute-bound   more FLOPs help
Inference, one token        memory-bound    more bandwidth helps
                                            batching helps most

You do not have to guess which one you have. Count the arithmetic and count the bytes, divide, and compare with the hardware's own ratio of peak arithmetic to peak bandwidth. Everything below the hardware's number is memory-bound.

operation                              FLOPs        bytes moved    intensity

add two n x n bf16 tensors             n^2          6 n^2          0.17
multiply two n x n bf16 matrices       2 n^3        6 n^2          n / 3
  n = 512                                                            171
  n = 4096                                                         1,365

A GPU with roughly 300 TFLOP/s and 2 TB/s balances at about 150 FLOP/byte.
Below that line, more arithmetic units buy you nothing at all.

The size of the gap is easy to underestimate until you measure it. There is no GPU here, so this is the same experiment on this machine's CPU, where the ratio is different but the physics is identical.

matmul  1024x1024    1,194 GFLOP/s     intensity 171 FLOP/byte
matmul  2048x2048      917 GFLOP/s     intensity 341 FLOP/byte
matmul  4096x4096      769 GFLOP/s     intensity 683 FLOP/byte
add     2048x2048        4.2 GFLOP/s   intensity 0.08          50.3 GB/s
add     4096x4096        4.0 GFLOP/s   intensity 0.08          48.3 GB/s
add     8192x8192        4.0 GFLOP/s   intensity 0.08          48.3 GB/s

Applying this to a language model gives the result people find counterintuitive. Generating one token for one sequence must read every weight in the model — 14 GB for a 7B model in bf16 — and does about two arithmetic operations per weight. Two FLOPs per two bytes is an intensity of 1, two orders of magnitude below the hardware's balance point, so the GPU's arithmetic units are idle more than 99 per cent of the time and generation speed is decided purely by how fast weights can be read.

Batch size fills the machine#

Small batches leave most of the GPU idle. Larger batches keep the cores busy, up to the limit of memory. For inference, batching many requests together is what turns a memory-bound workload into an efficient one.

The mechanism is worth spelling out because it explains why the effect is so large. The weights are read once per step regardless of how many sequences are in the batch. One sequence does one token's worth of arithmetic per weight read; thirty-two sequences do thirty-two tokens' worth for the same read. The arithmetic intensity rises in direct proportion to the batch size, and until it crosses the hardware's balance point the step takes very nearly the same wall-clock time.

batch   weight reads   tokens produced   intensity    time per step   tokens/s
   1      14 GB              1              1x            same           1x
   4      14 GB              4              4x            same           4x
  16      14 GB             16             16x            about same    ~16x
  64      14 GB             64             64x            rising         <64x
 256      14 GB            256            256x            compute-bound  flattens

Two costs push back. Latency for an individual request goes up, because a request waits for the batch to form and then shares the step with everyone else, so throughput and time-to-first-token trade against each other directly. And memory goes up: each sequence in the batch carries its own KV cache, which for a 7B model at 8,192 tokens of context is about 4 GiB per sequence with full multi-head attention. Sixteen concurrent sequences at that context need 64 GiB of cache against 12.6 GiB of weights, so the cache, not the model, is what fills the card.

Fusion#

Each separate operation reads its input from memory and writes its output back. Fusing several operations into one kernel keeps intermediate results on the chip, avoiding round trips. This is much of what compilers and optimised attention implementations do.

Count the traffic on a sequence you write every day. A linear layer followed by a bias add, a GELU and a dropout is four operations on the same tensor. Unfused, the intermediate result is written to memory and read back three times — six tensor-sized transfers that exist only because the operations were separate kernels. Fused, it is written once. The arithmetic is identical; roughly five-sixths of the memory traffic is gone, and since the sequence was memory-bound, that is very close to a five-fold speedup on that part of the network.

In practice you get this by asking rather than by writing kernels. torch.compile fuses elementwise chains automatically, and the attention implementations shipped with modern frameworks are already fused. The engineering skill is recognising which parts of your model are a chain of small elementwise operations, because those are the parts where a compiler has something to work with, and knowing that it will do nothing for a model that is already one large matrix multiplication.

When one GPU is not enough#

Before reaching for more hardware, exhaust the cheap moves, because they are usually enough and they are always faster to try. Mixed precision halves the activation memory and roughly doubles matrix multiply throughput. Gradient accumulation gets you a large effective batch with small resident activations. Activation checkpointing trades about a third more compute for a large memory saving. Each is one line and none of them requires a second device.

When you do add devices, the first thing to know is that the link between them is a hierarchy and it decides what is worth doing. Cards inside one machine talk over a fast direct interconnect at hundreds of gigabytes a second. Machines talk over a network, which is one or two orders of magnitude slower. A parallelism strategy that is fine inside a box can be unusable across boxes, and that boundary, not the number of GPUs, is the thing to design around.

What to skip: writing kernels. The skill that pays here is diagnosis — deciding which side of the roofline you are on, and choosing between a larger batch, a fused implementation, lower precision or more devices. Reading a kernel occasionally is useful; writing one is a specialisation very few roles ask for.

Before you can make GPU code faster you need honest numbers from it, and the obvious way to time it gives you the wrong ones: work is queued asynchronously, so a stopwatch around the call measures Python rather than the GPU. This section covers timing it properly, measuring peak memory, and the one ratio that explains most of what you will see, arithmetic intensity, the FLOPs done per byte moved, which says whether an operation is limited by the chip's maths or by its memory bandwidth. That ratio is why elementwise operations are slow, why fusing helps, and why generating tokens one at a time wastes a GPU until you batch.

The GPU runs ahead of Python#

import time, torch

x = torch.randn(8192, 8192, device="cuda", dtype=torch.bfloat16)
torch.cuda.synchronize()
start = time.perf_counter()
for _ in range(10):
    y = x @ x
torch.cuda.synchronize()                 # wait for the GPU to actually finish
elapsed = (time.perf_counter() - start) / 10
print(f"{2 * 8192 ** 3 / elapsed / 1e12:.0f} TFLOP/s achieved")

GPU operations are queued and return immediately. Without synchronising, you measure how fast Python queued the work, not how long it took. Warm up with a few iterations first, since the first call includes setup costs.

The error this produces is not a small one. A timing loop without a synchronise reports the cost of appending to a queue, which is microseconds, for work that takes milliseconds — so the number comes out hundreds of times too small and looks like a wonderful result. If you have ever measured a GPU operation and found it impossibly fast, this was why.

Three rules cover it. Synchronise before starting the clock and again before stopping it. Warm up first, because the first call to a given shape compiles kernels, allocates buffers and may run an autotuning pass. And repeat enough times that the measurement is not dominated by launch overhead, then divide. For per-kernel timing rather than wall-clock, CUDA events are the better tool because they are recorded on the device's own stream.

start, end = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)

for _ in range(5):                       # warm up: kernels, allocator, autotuning
    y = x @ x
torch.cuda.synchronize()

start.record()
for _ in range(50):
    y = x @ x
end.record()
torch.cuda.synchronize()
print(start.elapsed_time(end) / 50, "ms per matmul")

Memory#

torch.cuda.reset_peak_memory_stats()
train_step(batch)
print(torch.cuda.max_memory_allocated() / 2 ** 30, "GiB peak")

Four numbers exist and they mean different things, which is the source of a recurring confusion about why nvidia-smi and PyTorch disagree. memory_allocated is what your tensors currently hold. max_memory_allocated is the high-water mark of that, and it is the number that decides whether a larger batch will fit. memory_reserved is what PyTorch's caching allocator has taken from the driver and is holding on to for reuse, which is always at least the allocated figure and often much more. nvidia-smi reports the reserved figure plus the CUDA context, which is a gigabyte or so on its own.

So a process showing 40 GiB in nvidia-smi with 12 GiB of live tensors is not leaking. It is the allocator doing its job, and calling torch.cuda.empty_cache() to make the number look better will hand memory back to the driver and then slow you down reallocating it. The one case where it earns its place is before handing the card to another process or library that allocates outside PyTorch.

The high-water mark is almost always inside the backward pass, not the forward, because that is where activations and gradients coexist. Measuring around a full training step rather than a forward pass is therefore the only measurement that answers the question you are asking.

Compute-bound or memory-bound?#

arithmetic intensity = FLOPs performed / bytes moved to and from memory

Elementwise add of two 16-bit tensors:   1 FLOP per 6 bytes     -> about 0.17   memory-bound
Matrix multiply n x n, 16-bit:           2n^3 FLOPs / 6n^2 bytes -> n / 3
  n = 4096                              -> about 1,365               compute-bound

A GPU with ~300 TFLOP/s and ~2 TB/s balances at ~150 FLOPs per byte.
Below that, memory bandwidth limits speed; above, arithmetic does.

This is the roofline idea, and it explains most performance behaviour: why elementwise operations and small batches waste a GPU, why fusing operations helps, why token generation is memory-bound and why batching fixes it.

Sweep the batch size#

Measure samples per second and peak memory at batch sizes 1, 2, 4 and upwards until memory runs out. Throughput rises steeply and then flattens once the GPU is saturated. The knee of that curve is usually the batch size to use.

Choosing a GPU is four questions in a fixed order, and only the last one is about price. Does the job fit in the memory; is the work limited by bandwidth or by arithmetic; do the cards need to talk to each other; and only then, what does a unit of useful work cost. Taken in that order the decision is usually forced. Taken in the other order — starting from the hourly rate — you rent the cheapest card, discover the model does not fit, and pay for the week it takes to find out.

Memory first#

Memory is a hard constraint rather than a preference: below the threshold the job does not run at all, and above it the extra is worth nothing. So work out the requirement before looking at any other column. For inference the rule is roughly two bytes per parameter in bf16, plus the KV cache, which grows with batch and context. For training it is about sixteen bytes per parameter with mixed-precision Adam, plus activations — which is why a 7B model needs 104 GiB and will not train on a single 80 GiB card however cheap it is.

job                                   memory needed        smallest sensible card
7B inference, bf16, short context     ~15 GiB              24 GB (L4, A10G)
7B inference, long context, batched   ~30 GiB              40-80 GB
7B LoRA fine-tune                     ~25 GiB              40 GB
7B full fine-tune, Adam               ~104 GiB             multiple 80 GB
70B inference, 4-bit                  ~40 GiB              48-80 GB

Bandwidth for inference#

Generating one token reads every weight the model has and does a small amount of arithmetic with each. That makes single-stream generation memory-bound: the ceiling is how fast the weights can be moved out of memory, not how fast the chip can multiply. A 7B model in bf16 is about 14 GB, so a card with 2 TB/s can in principle read it around 140 times a second, and the achievable token rate follows from that number rather than from the TFLOPs.

Two consequences follow, and both are counterintuitive until the arithmetic is in front of you. First, for serving a single stream a card with more bandwidth beats a card with more compute, even at the same price. Second, batching is close to free on the compute side: a batch of sixteen reads the same weights once and does sixteen times the arithmetic, so throughput rises nearly linearly until the arithmetic finally becomes the limit. That is why inference services batch aggressively and why the per-request cost falls so sharply with load.

Interconnect for multi-GPU training#

The moment a job spans more than one card, what connects them matters more than the cards. The numbers section of the distributed training topic works this out: a 7B gradient sync is about 27 ms over NVLink and about 19.6 seconds over 10 Gb Ethernet — 6 per cent of a step against 98 per cent of it. Eight cards on a slow network is not a cluster, it is eight cards waiting.

Cost per useful unit#

Now, and only now, price. The rate per hour is not the cost of the work; the cost of the work is the rate divided by how much work the card does in that hour. Here is the same training job — ten exaFLOPs, a realistic forty per cent utilisation — priced across a range of cards.

# Approximate on-demand rates. These move constantly; the arithmetic does not.
gpus = [  # name,     VRAM, bf16 TFLOP/s, GB/s, $/hour
    ("T4",         16,   65,  300, 0.35),
    ("L4",         24,  121,  300, 0.80),
    ("A10G",       24,  125,  600, 1.00),
    ("A100 40GB",  40,  312, 1555, 2.50),
    ("A100 80GB",  80,  312, 2039, 3.50),
    ("H100 80GB",  80,  989, 3350, 6.00),
]

for name, vram, tflops, bw, price in gpus:
    hours = 1e19 / (tflops * 1e12 * 0.4) / 3600      # 1e19 FLOPs at 40% MFU
    fits = "" if vram >= 80 else "   <- 7B + Adam does not fit"
    cost = hours * price
    print(f"{name:11} {hours:6.1f} h   " "$" f"{cost:7.2f}{fits}")

# T4          106.8 h   $  37.39   <- 7B + Adam does not fit
# L4           57.4 h   $  45.91   <- 7B + Adam does not fit
# A10G         55.6 h   $  55.56   <- 7B + Adam does not fit
# A100 40GB    22.3 h   $  55.64   <- 7B + Adam does not fit
# A100 80GB    22.3 h   $  77.90
# H100 80GB     7.0 h   $  42.13

The T4 row is the trap in the other direction. It shows the lowest total, and it cannot hold the model, so that figure is for a job it will never run. Apply the memory filter first and then compare cost, or you will reason carefully about a column that does not apply. The same table also shows why the A100 80GB is often the worst purchase on the list for compute-heavy work: it is priced for its memory and has the same arithmetic throughput as the 40GB card.

Spot instances and idle GPUs#

Two things dominate a real GPU bill and neither is the hourly rate. The first is spot capacity: interruptible instances are typically sixty to ninety per cent cheaper, and they are entirely usable for training that checkpoints properly — which is the same discipline a long run needs anyway. If your job cannot survive being killed, fixing that is worth more than any negotiation on price.

The second is idle time, which is where most money actually goes. A development instance left running overnight bills identically to one doing useful work, and a notebook attached to an A100 for a week of intermittent typing is the single most common line item on a surprising bill. The defences are unglamorous and effective.

• Put an idle shutdown on every interactive instance. Most platforms have one; it is off by default.
• Develop on the smallest card that runs the code, and move to the large one only for the real run.
• Tag everything by owner and job, and look at the bill weekly. Untagged spend is nobody's and grows.
• Prefer a job that exits to a session you leave open. A queued batch job cannot be forgotten.
• Check utilisation before renting more: a card at 30 per cent is a data-loading problem, and a second card will also sit at 30 per cent.

That last one is the whole topic in a sentence. More hardware multiplies whatever efficiency you already have, so a job running at thirty per cent utilisation on one card runs at thirty per cent on eight, for eight times the money. Measure first.