What makes inference expensive

Running a model on your own hardware is priced quite differently from calling an API, and the arithmetic is not obvious: what limits speed is how fast the weights can be read out of memory, not how fast the chip can multiply. This section works through that and the levers it exposes, from lower-precision weights to reusing computation between tokens to serving many requests at once. It is what lets you answer 'should we host this ourselves?' with numbers rather than an instinct, and the numbers usually say no.

Bandwidth, not compute#

Generating a token requires reading every weight in the model from memory. For a large model that is tens of gigabytes per token. The GPU spends most of its time waiting for memory rather than calculating, which is why memory bandwidth predicts speed better than raw compute does.

70B parameters at 16 bits  = about 140 GB read per token
Quantized to 4 bits        = about 35 GB read per token

Four times less to move, so roughly four times faster.

The way to make that precise, and the form an interviewer is usually fishing for, is arithmetic intensity: how many floating point operations you perform for each byte you move. One decode step at batch size B on a model of P parameters stored at w bytes each does about 2PB operations, because every weight is used once per sequence in the batch, and reads about Pw bytes, because the weights are read once for the whole batch however many sequences are in it. Divide one by the other and the intensity is 2B over w. The model size cancels. A 7B model and a 700B model at the same batch size and precision sit at exactly the same point on the roofline.

Every accelerator has its own ratio of peak arithmetic to peak bandwidth, and that ratio is the line you have to cross before more compute helps you at all.

# One decode step, batch B, P parameters stored at w bytes each:
#   arithmetic  = 2 * P * B FLOPs     every weight used once per sequence
#   memory read = P * w bytes         the weights are read once for the whole batch
#   intensity   = 2B / w FLOP per byte -- it does not depend on P at all.

cards = [("H100 SXM", 989e12, 3.35e12),
         ("A100 80GB", 312e12, 2.04e12),
         ("L40S", 362e12, 0.864e12)]

print(f"{'card':11}{'TFLOP/s':>9}{'TB/s':>7}{'FLOP/byte':>11}"
      f"{'break-even B':>14}{'at 4-bit':>10}")
for name, flops, bw in cards:
    ratio = flops / bw
    print(f"{name:11}{flops/1e12:9.0f}{bw/1e12:7.2f}{ratio:11.0f}"
          f"{ratio * 2 / 2:14.0f}{ratio * 0.5 / 2:10.0f}")

# card         TFLOP/s   TB/s  FLOP/byte  break-even B  at 4-bit
# H100 SXM         989   3.35        295           295        74
# A100 80GB        312   2.04        153           153        38
# L40S             362   0.86        419           419       105

Two readings of that table are worth carrying. Serving one user at a time on an H100 uses roughly one three-hundredth of the arithmetic the card can do, so the phrase 'we need a faster GPU' is almost always the wrong diagnosis: you need a fuller batch. And quantizing moves the break-even down rather than up, from 295 to 74 on the same card, because fewer weight bytes means less to hide behind. Quantization makes a single stream faster and makes it easier to run out of compute later, which is not a contradiction but does surprise people.

The absolute floor on single-stream speed follows from the same arithmetic and is worked in the faster-generation section of this topic, along with the serving techniques that chase the batch size upwards. This section stays on what the numbers mean for a decision to host.

Quantization#

Weights are stored at lower precision: 8 bit or 4 bit instead of 16. The model uses less memory and runs faster, and accuracy degrades, usually slightly at 8 bit and noticeably at 4 bit depending on the model and the task.

• 16 bit: the baseline.
• 8 bit: roughly half the memory, quality loss usually small enough to ignore.
• 4 bit: roughly a quarter, quality loss real and task-dependent. Measure it.
• Below 4 bit: degrades quickly outside specialised methods.

What actually happens to a weight is worth knowing, because it explains both why the ladder behaves the way it does and why the formats have the names they do. Weights are split into small groups, commonly 128 of them from one row. For each group, the smallest and largest value are recorded and the range between them is cut into equal steps, so each weight becomes a small integer saying which step it is nearest. The group's two endpoints are kept at full precision alongside the integers, and the kernel reconstructs an approximate float from the integer and the endpoints as it reads. Nothing is learned and nothing is discarded selectively; it is rounding, applied per group so that one loud group cannot ruin a quiet one.

import numpy as np

rng = np.random.default_rng(0)

def quantize(W, bits, group=128):
    """Affine per-group quantization: the shape almost every 4-bit format uses."""
    g = W.reshape(-1, group)
    lo, hi = g.min(axis=1, keepdims=True), g.max(axis=1, keepdims=True)
    scale = (hi - lo) / (2 ** bits - 1)
    q = np.rint((g - lo) / scale)                 # the integers actually stored
    return (q * scale + lo).reshape(W.shape)      # what the kernel reconstructs

W = (rng.standard_normal((4096, 4096)) * 0.02).astype(np.float32)
outlier = W.copy()
idx = rng.choice(W.size, 4096, replace=False)     # 0.02% of weights, 20x larger
outlier.flat[idx] *= 20

x = rng.standard_normal(4096).astype(np.float32)
print(f"{'bits':>5}{'bits/weight':>13}{'well-behaved':>14}{'with outliers':>15}")
for bits in (8, 4, 3, 2):
    effective = bits + 32 / 128                   # one fp16 scale and zero per 128
    errs = []
    for M in (W, outlier):
        ref = x @ M
        errs.append(np.linalg.norm(x @ quantize(M, bits) - ref) / np.linalg.norm(ref))
    print(f"{bits:5d}{effective:13.2f}{errs[0]:14.4f}{errs[1]:15.4f}")

#  bits  bits/weight  well-behaved  with outliers
#     8         8.25        0.0060         0.0069
#     4         4.25        0.1001         0.1167
#     3         3.25        0.2161         0.2680
#     2         2.25        0.5039         0.5707

The bits-per-weight column is the one people forget when sizing a machine. A 4-bit format is not 4 bits per weight: the group endpoints cost 32 bits per 128 weights, which is another quarter of a bit each, so it is 4.25 and a smaller group size makes it worse. Quoting 4-bit as exactly a quarter of 16-bit will leave you a few per cent short of the memory you budgeted, which matters precisely when you are trying to fit a model on one card.

The outlier column barely moves, and that is misleading in an instructive way. Per-group quantization confines the damage to the groups that contain an outlier, so a sparse scattering of them is almost invisible in the aggregate while being severe where it lands.

# Where the 4-bit damage actually lands: per group of 128 weights.
group = 128
for name, M in (("well-behaved", W), ("with outliers", outlier)):
    g = M.reshape(-1, group)
    err = np.abs(quantize(M, 4).reshape(-1, group) - g).mean(axis=1) / np.abs(g).mean(axis=1)
    hit = np.zeros(g.shape[0], dtype=bool)
    hit[np.unique(idx // group)] = True
    print(f"{name:14} all groups {err.mean():.4f}"
          f"   groups holding an outlier {err[hit].mean():.4f}"
          f"   the rest {err[~hit].mean():.4f}")
print(f"{(np.unique(idx // group).size / (W.size // group)):.1%} of groups contain an outlier")

# well-behaved   all groups 0.1070   groups holding an outlier 0.1070   the rest 0.1070
# with outliers  all groups 0.1134   groups holding an outlier 0.3169   the rest 0.1070
# 3.1% of groups contain an outlier

Real transformer weights and activations do have outlier channels, and that concentration is the reason the serious methods exist. Keeping a small set of salient channels at higher precision, or calibrating the rounding against real activations so the error that matters is minimised rather than the error in the weights, is what separates a usable 4-bit model from one that is merely small. It is also why the same nominal precision from two different tools can behave very differently, and why the aggregate error above is a shape rather than a prediction.

KV cache#

During generation the model would otherwise recompute the attention keys and values for every previous token at every step. Caching them turns that repeated work into a lookup. It is why generation is not far slower than it is, and the cache is a major consumer of memory on long sequences.

The size of that cache is arithmetic you should be able to do in your head, because it decides how many conversations fit on a card and therefore what the thing costs to run. Per token it is two, for keys and values, times the layer count, times the number of key and value heads, times the head dimension, times the bytes per element. Note the key and value head count rather than the query head count: grouped-query attention, covered in the modern-variants section of the transformers topic, is the change that divides this figure by four on most current models.

GiB = 1024 ** 3
P, layers, kv_heads, head_dim = 70e9, 80, 8, 128     # a 70B grouped-query model
CARD = 80 * GiB
OVERHEAD = 4 * GiB                                   # activations, workspace, fragmentation

per_token = 2 * layers * kv_heads * head_dim * 2     # keys and values, 16-bit
print(f"KV cache: {per_token/1024:.0f} KiB per token, "
      f"{per_token*8192/GiB:.2f} GiB for one 8k conversation")

print(f"\n{'weights':>10}{'weight GiB':>12}{'cache GiB':>11}"
      f"{'seqs at 2k':>12}{'at 8k':>8}{'at 32k':>8}")
for bits in (16, 8, 4):
    wbytes = P * bits / 8
    free = CARD - wbytes - OVERHEAD
    row = [int(max(0, free) // (per_token * n)) for n in (2048, 8192, 32768)]
    print(f"{str(bits)+'-bit':>10}{wbytes/GiB:12.1f}{free/GiB:11.1f}"
          f"{row[0]:12d}{row[1]:8d}{row[2]:8d}")

# KV cache: 320 KiB per token, 2.50 GiB for one 8k conversation
#
#    weights  weight GiB  cache GiB  seqs at 2k   at 8k  at 32k
#     16-bit       130.4      -54.4           0       0       0
#      8-bit        65.2       10.8          17       4       1
#      4-bit        32.6       43.4          69      17       4

This is the table that turns quantization from a speed trick into the thing that decides whether a deployment exists. At 16-bit the model does not load. At 8-bit it loads and leaves room for four concurrent 8k conversations. At 4-bit it leaves room for seventeen. The weights got four times smaller and the useful concurrency went up by more than four, because every byte the weights give back is a byte the cache can use, and the cache is what concurrency is actually made of.

It also explains the context-length line on a hosting page. The same card holds sixty-nine 2k conversations or four 32k ones, a seventeen-fold difference in how many customers a machine serves, which is why long context is priced the way it is even though the per-token arithmetic barely changes.

Batching#

Since the bottleneck is reading weights, serving several requests together costs barely more than serving one: the weights are read once and used for all of them. This is the main reason hosted APIs are cheap and why a self-hosted model serving one user at a time is very poor value.

Put the two levers together and you can price a card. Take the 70B model at 4-bit from the table above, work out how long one decode step takes at each batch size as the larger of the weight read and the arithmetic, and divide the rent by the tokens it produces.

FLOPS, BW = 989e12, 3.35e12      # H100 SXM published peaks
EFF = 0.7                        # no kernel reaches peak; 70% is a fair ceiling
HOURLY = 2.50                    # USD per hour for one card; substitute your quote

wbytes = P * 4 / 8               # 4-bit weights
print(f"{'batch':>7}{'ms/step':>9}{'tok/s total':>13}{'tok/s each':>12}{'USD / 1M':>10}")
for B in (1, 4, 16, 17, 64, 74, 256):
    step = max(wbytes / (BW * EFF), 2 * P * B / (FLOPS * EFF))
    tps = B / step
    print(f"{B:7d}{step*1000:9.2f}{tps:13.0f}{tps/B:12.1f}{HOURLY/3600/tps*1e6:10.3f}")

#   batch  ms/step  tok/s total  tok/s each  USD / 1M
#       1    14.93           67        67.0    10.365
#       4    14.93          268        67.0     2.591
#      16    14.93         1072        67.0     0.648
#      17    14.93         1139        67.0     0.610
#      64    14.93         4288        67.0     0.162
#      74    14.96         4945        66.8     0.140
#     256    51.77         4945        19.3     0.140

Three things in that table decide everything. Up to the break-even batch of 74, adding a sequence is free: the step still takes 14.93 ms, each sequence still gets 67 tokens a second, and the unit cost falls in exact proportion to the batch. Past it the step lengthens and the per-sequence rate collapses, so batch 256 produces the same 4,945 tokens a second as batch 74 while making every individual user wait nearly four times longer. And the cost per million tokens spans a factor of seventy-four across the same hardware and the same model, decided entirely by how full the batch is.

Cross that against the cache table and the real constraint appears. Batch 74 needs 74 caches resident, which at 8k tokens each is 185 GiB and does not fit; the card holds seventeen. So the reachable row for ordinary 8k interactive traffic is batch 17 at 0.61 USD per million, not the 0.14 the roofline would allow. Memory stops you before arithmetic does, which is why paged allocation and cache quantization matter so much in practice, and those are covered in the faster-generation section of this topic.

When self-hosting is worth it#

• Data cannot leave your infrastructure. This is the reason that actually justifies it most often.
• Very high, steady volume, where batching keeps the hardware busy.
• A fine-tuned model that no provider hosts.
• Latency requirements that a network round trip cannot meet.

Otherwise the arithmetic rarely favours it. An idle GPU costs the same as a busy one, and providers batch across many customers in a way a single tenant cannot.

Price it rather than asserting it. The card is rented by the hour whether or not anything is running, so the only two numbers that matter are the batch size you actually sustain and the fraction of the month the card is busy. Real interactive traffic has evenings, weekends and a working week, so a duty cycle of ten to thirty per cent is what a single tenant should expect unless there is a queue behind the endpoint.

HOURS = 24 * 30
UPKEEP = 4000.0                  # a quarter of an engineer on serving, scaling, upgrades
card_cost = HOURLY * HOURS
print(f"one card: {card_cost:,.0f} USD a month in rent, plus {UPKEEP:,.0f} in upkeep\n")

duties = (0.10, 0.30, 0.60, 1.00)
print(f"{'batch':>6}{'tok/s':>8}" + "".join(f"{f'duty {d:.0%}':>11}" for d in duties))
for B in (1, 17, 74):
    step = max(wbytes / (BW * EFF), 2 * P * B / (FLOPS * EFF))
    tps = B / step
    print(f"{B:6d}{tps:8.0f}"
          + "".join(f"{card_cost / (tps * 3600 * HOURS * d / 1e6):11.2f}" for d in duties))
print("\nUSD per million tokens, rent only. Upkeep does not fall with volume.")

# one card: 1,800 USD a month in rent, plus 4,000 in upkeep
#
#  batch   tok/s   duty 10%   duty 30%   duty 60%  duty 100%
#      1      67     103.65      34.55      17.27      10.36
#     17    1139       6.10       2.03       1.02       0.61
#     74    4945       1.40       0.47       0.23       0.14
#
# USD per million tokens, rent only. Upkeep does not fall with volume.

Read the cell you are actually in. A team hosting a 70B model for internal use, which is one card, ordinary office-hours traffic and a batch capped at seventeen by the cache, is at 2.03 USD per million tokens in rent, plus 4,000 a month in upkeep that does not fall with volume, plus whatever the build cost. That is the number to compare against a provider's price list, and against a provider's price list it usually loses. The bottom-right cell, 0.14, is the one people quote, and reaching it needs a full batch of 74 running around the clock, which at 8k context needs more memory than the card has.

That gap is not a failure of your engineering. A provider runs a queue deep enough to keep the batch full, spreads it across customers whose peaks do not coincide, amortises the same weights over far more traffic, and employs people to do nothing else. A single tenant with spiky traffic cannot reproduce any of those four. So the honest framing for an interview is that self-hosting buys control, data residency, a model nobody else will serve, and predictable latency, and that if the argument being made for it is unit cost, the arithmetic above is how you check the claim before anyone signs for hardware.

One more line belongs in the comparison and is routinely left out. An hourly rate is the on-demand price; a reservation is cheaper and commits you for a year, which is the opposite of what an uncertain workload wants. Engineering time is real, recurring and larger than people estimate, because serving is not a thing you set up once: models are replaced, engines gain features you need, and someone is on call for it. Count that before the comparison, not after.