Quantization & inference optimisation

Running a model on your own hardware is priced quite differently from calling an API, and the arithmetic is not obvious: what limits speed is how fast the weights can be read out of memory, not how fast the chip can multiply. This section works through that and the levers it exposes, from lower-precision weights to reusing computation between tokens to serving many requests at once. It is what lets you answer 'should we host this ourselves?' with numbers rather than an instinct, and the numbers usually say no.

Bandwidth, not compute#

Generating a token requires reading every weight in the model from memory. For a large model that is tens of gigabytes per token. The GPU spends most of its time waiting for memory rather than calculating, which is why memory bandwidth predicts speed better than raw compute does.

70B parameters at 16 bits  = about 140 GB read per token
Quantized to 4 bits        = about 35 GB read per token

Four times less to move, so roughly four times faster.

The way to make that precise, and the form an interviewer is usually fishing for, is arithmetic intensity: how many floating point operations you perform for each byte you move. One decode step at batch size B on a model of P parameters stored at w bytes each does about 2PB operations, because every weight is used once per sequence in the batch, and reads about Pw bytes, because the weights are read once for the whole batch however many sequences are in it. Divide one by the other and the intensity is 2B over w. The model size cancels. A 7B model and a 700B model at the same batch size and precision sit at exactly the same point on the roofline.

Every accelerator has its own ratio of peak arithmetic to peak bandwidth, and that ratio is the line you have to cross before more compute helps you at all.

# One decode step, batch B, P parameters stored at w bytes each:
#   arithmetic  = 2 * P * B FLOPs     every weight used once per sequence
#   memory read = P * w bytes         the weights are read once for the whole batch
#   intensity   = 2B / w FLOP per byte -- it does not depend on P at all.

cards = [("H100 SXM", 989e12, 3.35e12),
         ("A100 80GB", 312e12, 2.04e12),
         ("L40S", 362e12, 0.864e12)]

print(f"{'card':11}{'TFLOP/s':>9}{'TB/s':>7}{'FLOP/byte':>11}"
      f"{'break-even B':>14}{'at 4-bit':>10}")
for name, flops, bw in cards:
    ratio = flops / bw
    print(f"{name:11}{flops/1e12:9.0f}{bw/1e12:7.2f}{ratio:11.0f}"
          f"{ratio * 2 / 2:14.0f}{ratio * 0.5 / 2:10.0f}")

# card         TFLOP/s   TB/s  FLOP/byte  break-even B  at 4-bit
# H100 SXM         989   3.35        295           295        74
# A100 80GB        312   2.04        153           153        38
# L40S             362   0.86        419           419       105

Two readings of that table are worth carrying. Serving one user at a time on an H100 uses roughly one three-hundredth of the arithmetic the card can do, so the phrase 'we need a faster GPU' is almost always the wrong diagnosis: you need a fuller batch. And quantizing moves the break-even down rather than up, from 295 to 74 on the same card, because fewer weight bytes means less to hide behind. Quantization makes a single stream faster and makes it easier to run out of compute later, which is not a contradiction but does surprise people.

The absolute floor on single-stream speed follows from the same arithmetic and is worked in the faster-generation section of this topic, along with the serving techniques that chase the batch size upwards. This section stays on what the numbers mean for a decision to host.

Quantization#

Weights are stored at lower precision: 8 bit or 4 bit instead of 16. The model uses less memory and runs faster, and accuracy degrades, usually slightly at 8 bit and noticeably at 4 bit depending on the model and the task.

• 16 bit: the baseline.
• 8 bit: roughly half the memory, quality loss usually small enough to ignore.
• 4 bit: roughly a quarter, quality loss real and task-dependent. Measure it.
• Below 4 bit: degrades quickly outside specialised methods.

What actually happens to a weight is worth knowing, because it explains both why the ladder behaves the way it does and why the formats have the names they do. Weights are split into small groups, commonly 128 of them from one row. For each group, the smallest and largest value are recorded and the range between them is cut into equal steps, so each weight becomes a small integer saying which step it is nearest. The group's two endpoints are kept at full precision alongside the integers, and the kernel reconstructs an approximate float from the integer and the endpoints as it reads. Nothing is learned and nothing is discarded selectively; it is rounding, applied per group so that one loud group cannot ruin a quiet one.

import numpy as np

rng = np.random.default_rng(0)

def quantize(W, bits, group=128):
    """Affine per-group quantization: the shape almost every 4-bit format uses."""
    g = W.reshape(-1, group)
    lo, hi = g.min(axis=1, keepdims=True), g.max(axis=1, keepdims=True)
    scale = (hi - lo) / (2 ** bits - 1)
    q = np.rint((g - lo) / scale)                 # the integers actually stored
    return (q * scale + lo).reshape(W.shape)      # what the kernel reconstructs

W = (rng.standard_normal((4096, 4096)) * 0.02).astype(np.float32)
outlier = W.copy()
idx = rng.choice(W.size, 4096, replace=False)     # 0.02% of weights, 20x larger
outlier.flat[idx] *= 20

x = rng.standard_normal(4096).astype(np.float32)
print(f"{'bits':>5}{'bits/weight':>13}{'well-behaved':>14}{'with outliers':>15}")
for bits in (8, 4, 3, 2):
    effective = bits + 32 / 128                   # one fp16 scale and zero per 128
    errs = []
    for M in (W, outlier):
        ref = x @ M
        errs.append(np.linalg.norm(x @ quantize(M, bits) - ref) / np.linalg.norm(ref))
    print(f"{bits:5d}{effective:13.2f}{errs[0]:14.4f}{errs[1]:15.4f}")

#  bits  bits/weight  well-behaved  with outliers
#     8         8.25        0.0060         0.0069
#     4         4.25        0.1001         0.1167
#     3         3.25        0.2161         0.2680
#     2         2.25        0.5039         0.5707

The bits-per-weight column is the one people forget when sizing a machine. A 4-bit format is not 4 bits per weight: the group endpoints cost 32 bits per 128 weights, which is another quarter of a bit each, so it is 4.25 and a smaller group size makes it worse. Quoting 4-bit as exactly a quarter of 16-bit will leave you a few per cent short of the memory you budgeted, which matters precisely when you are trying to fit a model on one card.

The outlier column barely moves, and that is misleading in an instructive way. Per-group quantization confines the damage to the groups that contain an outlier, so a sparse scattering of them is almost invisible in the aggregate while being severe where it lands.

# Where the 4-bit damage actually lands: per group of 128 weights.
group = 128
for name, M in (("well-behaved", W), ("with outliers", outlier)):
    g = M.reshape(-1, group)
    err = np.abs(quantize(M, 4).reshape(-1, group) - g).mean(axis=1) / np.abs(g).mean(axis=1)
    hit = np.zeros(g.shape[0], dtype=bool)
    hit[np.unique(idx // group)] = True
    print(f"{name:14} all groups {err.mean():.4f}"
          f"   groups holding an outlier {err[hit].mean():.4f}"
          f"   the rest {err[~hit].mean():.4f}")
print(f"{(np.unique(idx // group).size / (W.size // group)):.1%} of groups contain an outlier")

# well-behaved   all groups 0.1070   groups holding an outlier 0.1070   the rest 0.1070
# with outliers  all groups 0.1134   groups holding an outlier 0.3169   the rest 0.1070
# 3.1% of groups contain an outlier

Real transformer weights and activations do have outlier channels, and that concentration is the reason the serious methods exist. Keeping a small set of salient channels at higher precision, or calibrating the rounding against real activations so the error that matters is minimised rather than the error in the weights, is what separates a usable 4-bit model from one that is merely small. It is also why the same nominal precision from two different tools can behave very differently, and why the aggregate error above is a shape rather than a prediction.

KV cache#

During generation the model would otherwise recompute the attention keys and values for every previous token at every step. Caching them turns that repeated work into a lookup. It is why generation is not far slower than it is, and the cache is a major consumer of memory on long sequences.

The size of that cache is arithmetic you should be able to do in your head, because it decides how many conversations fit on a card and therefore what the thing costs to run. Per token it is two, for keys and values, times the layer count, times the number of key and value heads, times the head dimension, times the bytes per element. Note the key and value head count rather than the query head count: grouped-query attention, covered in the modern-variants section of the transformers topic, is the change that divides this figure by four on most current models.

GiB = 1024 ** 3
P, layers, kv_heads, head_dim = 70e9, 80, 8, 128     # a 70B grouped-query model
CARD = 80 * GiB
OVERHEAD = 4 * GiB                                   # activations, workspace, fragmentation

per_token = 2 * layers * kv_heads * head_dim * 2     # keys and values, 16-bit
print(f"KV cache: {per_token/1024:.0f} KiB per token, "
      f"{per_token*8192/GiB:.2f} GiB for one 8k conversation")

print(f"\n{'weights':>10}{'weight GiB':>12}{'cache GiB':>11}"
      f"{'seqs at 2k':>12}{'at 8k':>8}{'at 32k':>8}")
for bits in (16, 8, 4):
    wbytes = P * bits / 8
    free = CARD - wbytes - OVERHEAD
    row = [int(max(0, free) // (per_token * n)) for n in (2048, 8192, 32768)]
    print(f"{str(bits)+'-bit':>10}{wbytes/GiB:12.1f}{free/GiB:11.1f}"
          f"{row[0]:12d}{row[1]:8d}{row[2]:8d}")

# KV cache: 320 KiB per token, 2.50 GiB for one 8k conversation
#
#    weights  weight GiB  cache GiB  seqs at 2k   at 8k  at 32k
#     16-bit       130.4      -54.4           0       0       0
#      8-bit        65.2       10.8          17       4       1
#      4-bit        32.6       43.4          69      17       4

This is the table that turns quantization from a speed trick into the thing that decides whether a deployment exists. At 16-bit the model does not load. At 8-bit it loads and leaves room for four concurrent 8k conversations. At 4-bit it leaves room for seventeen. The weights got four times smaller and the useful concurrency went up by more than four, because every byte the weights give back is a byte the cache can use, and the cache is what concurrency is actually made of.

It also explains the context-length line on a hosting page. The same card holds sixty-nine 2k conversations or four 32k ones, a seventeen-fold difference in how many customers a machine serves, which is why long context is priced the way it is even though the per-token arithmetic barely changes.

Batching#

Since the bottleneck is reading weights, serving several requests together costs barely more than serving one: the weights are read once and used for all of them. This is the main reason hosted APIs are cheap and why a self-hosted model serving one user at a time is very poor value.

Put the two levers together and you can price a card. Take the 70B model at 4-bit from the table above, work out how long one decode step takes at each batch size as the larger of the weight read and the arithmetic, and divide the rent by the tokens it produces.

FLOPS, BW = 989e12, 3.35e12      # H100 SXM published peaks
EFF = 0.7                        # no kernel reaches peak; 70% is a fair ceiling
HOURLY = 2.50                    # USD per hour for one card; substitute your quote

wbytes = P * 4 / 8               # 4-bit weights
print(f"{'batch':>7}{'ms/step':>9}{'tok/s total':>13}{'tok/s each':>12}{'USD / 1M':>10}")
for B in (1, 4, 16, 17, 64, 74, 256):
    step = max(wbytes / (BW * EFF), 2 * P * B / (FLOPS * EFF))
    tps = B / step
    print(f"{B:7d}{step*1000:9.2f}{tps:13.0f}{tps/B:12.1f}{HOURLY/3600/tps*1e6:10.3f}")

#   batch  ms/step  tok/s total  tok/s each  USD / 1M
#       1    14.93           67        67.0    10.365
#       4    14.93          268        67.0     2.591
#      16    14.93         1072        67.0     0.648
#      17    14.93         1139        67.0     0.610
#      64    14.93         4288        67.0     0.162
#      74    14.96         4945        66.8     0.140
#     256    51.77         4945        19.3     0.140

Three things in that table decide everything. Up to the break-even batch of 74, adding a sequence is free: the step still takes 14.93 ms, each sequence still gets 67 tokens a second, and the unit cost falls in exact proportion to the batch. Past it the step lengthens and the per-sequence rate collapses, so batch 256 produces the same 4,945 tokens a second as batch 74 while making every individual user wait nearly four times longer. And the cost per million tokens spans a factor of seventy-four across the same hardware and the same model, decided entirely by how full the batch is.

Cross that against the cache table and the real constraint appears. Batch 74 needs 74 caches resident, which at 8k tokens each is 185 GiB and does not fit; the card holds seventeen. So the reachable row for ordinary 8k interactive traffic is batch 17 at 0.61 USD per million, not the 0.14 the roofline would allow. Memory stops you before arithmetic does, which is why paged allocation and cache quantization matter so much in practice, and those are covered in the faster-generation section of this topic.

When self-hosting is worth it#

• Data cannot leave your infrastructure. This is the reason that actually justifies it most often.
• Very high, steady volume, where batching keeps the hardware busy.
• A fine-tuned model that no provider hosts.
• Latency requirements that a network round trip cannot meet.

Otherwise the arithmetic rarely favours it. An idle GPU costs the same as a busy one, and providers batch across many customers in a way a single tenant cannot.

Price it rather than asserting it. The card is rented by the hour whether or not anything is running, so the only two numbers that matter are the batch size you actually sustain and the fraction of the month the card is busy. Real interactive traffic has evenings, weekends and a working week, so a duty cycle of ten to thirty per cent is what a single tenant should expect unless there is a queue behind the endpoint.

HOURS = 24 * 30
UPKEEP = 4000.0                  # a quarter of an engineer on serving, scaling, upgrades
card_cost = HOURLY * HOURS
print(f"one card: {card_cost:,.0f} USD a month in rent, plus {UPKEEP:,.0f} in upkeep\n")

duties = (0.10, 0.30, 0.60, 1.00)
print(f"{'batch':>6}{'tok/s':>8}" + "".join(f"{f'duty {d:.0%}':>11}" for d in duties))
for B in (1, 17, 74):
    step = max(wbytes / (BW * EFF), 2 * P * B / (FLOPS * EFF))
    tps = B / step
    print(f"{B:6d}{tps:8.0f}"
          + "".join(f"{card_cost / (tps * 3600 * HOURS * d / 1e6):11.2f}" for d in duties))
print("\nUSD per million tokens, rent only. Upkeep does not fall with volume.")

# one card: 1,800 USD a month in rent, plus 4,000 in upkeep
#
#  batch   tok/s   duty 10%   duty 30%   duty 60%  duty 100%
#      1      67     103.65      34.55      17.27      10.36
#     17    1139       6.10       2.03       1.02       0.61
#     74    4945       1.40       0.47       0.23       0.14
#
# USD per million tokens, rent only. Upkeep does not fall with volume.

Read the cell you are actually in. A team hosting a 70B model for internal use, which is one card, ordinary office-hours traffic and a batch capped at seventeen by the cache, is at 2.03 USD per million tokens in rent, plus 4,000 a month in upkeep that does not fall with volume, plus whatever the build cost. That is the number to compare against a provider's price list, and against a provider's price list it usually loses. The bottom-right cell, 0.14, is the one people quote, and reaching it needs a full batch of 74 running around the clock, which at 8k context needs more memory than the card has.

That gap is not a failure of your engineering. A provider runs a queue deep enough to keep the batch full, spreads it across customers whose peaks do not coincide, amortises the same weights over far more traffic, and employs people to do nothing else. A single tenant with spiky traffic cannot reproduce any of those four. So the honest framing for an interview is that self-hosting buys control, data residency, a model nobody else will serve, and predictable latency, and that if the argument being made for it is unit cost, the arithmetic above is how you check the claim before anyone signs for hardware.

One more line belongs in the comparison and is routinely left out. An hourly rate is the on-demand price; a reservation is cheaper and commits you for a year, which is the opposite of what an uncertain workload wants. Engineering time is real, recurring and larger than people estimate, because serving is not a thing you set up once: models are replaced, engines gain features you need, and someone is on call for it. Count that before the comparison, not after.

Whether a model fits on a given GPU is arithmetic you can do in a minute, and you will be asked to. This section is that arithmetic: parameters times bytes per parameter for the weights, and then the KV cache, which is charged per token and per sequence and is usually what runs out first when serving several users at long contexts. The second half is practical: quantizing a matrix by hand so you can see exactly what rounding costs, running that on a trained model, and reading the result, since the damage lands on particular abilities rather than showing up in casual chat.

Weights#

8 billion parameters
  16-bit (2 bytes each)   16 GB
  8-bit  (1 byte)          8 GB
  4-bit  (half a byte)     4 GB   plus a little for scales and some layers kept at higher precision

Parameters times bytes per parameter, and nothing else. The only subtlety is that a 4-bit checkpoint is never exactly half a byte per weight: every group of weights carries a scale, usually at 16 bits, and most methods keep a few sensitive tensors at higher precision. Budget five to ten per cent above the arithmetic and you will be close.

The KV cache, which people forget#

per token = 2 (key and value) x layers x kv_heads x head_dim x bytes

Example: 32 layers, 8 KV heads, head_dim 128, 16-bit
  2 x 32 x 8 x 128 x 2 bytes = 131,072 bytes = 128 KiB per token
  8,000-token context        = about 1 GiB per sequence
  batch of 16 at 8k tokens   = about 16 GiB, more than the 4-bit weights

For serving, the cache often decides how many requests fit, not the weights. That is why techniques that share key and value heads, and quantizing the cache itself, matter so much. Note which numbers appear and which do not: the number of query heads is irrelevant, because grouped-query attention shares one key and value head across several query heads, and only the shared ones are stored.

Does it fit, as a function rather than an argument#

This is the question an interviewer asks and the question you will answer weekly on the job, so write it down once. The example below is an 8B model on a 24 GB card, serving sixteen users at an 8k context, and it runs through the options in the order you would actually try them.

def fits(params_b, layers, kv_heads, head_dim, weight_bits, kv_bits,
         batch, ctx, gpu_gib):
    weights = params_b * 1e9 * weight_bits / 8
    per_token = 2 * layers * kv_heads * head_dim * kv_bits / 8
    cache = per_token * batch * ctx
    total = (weights + cache) / 2**30
    print("%2d-bit weights, %2d-bit cache, batch %2d at %5d ctx: "
          "weights %5.1f + cache %5.1f = %5.1f GiB  %s"
          % (weight_bits, kv_bits, batch, ctx, weights / 2**30,
             cache / 2**30, total, "fits" if total < gpu_gib else "does not fit"))

for wb, kb, batch in ((16, 16, 16), (8, 16, 16), (4, 16, 16), (4, 8, 16), (4, 8, 32)):
    fits(8, 32, 8, 128, wb, kb, batch, 8192, gpu_gib=24)

# 16-bit weights, 16-bit cache, batch 16 at  8192 ctx: weights  14.9 + cache  16.0 =  30.9 GiB  does not fit
#  8-bit weights, 16-bit cache, batch 16 at  8192 ctx: weights   7.5 + cache  16.0 =  23.5 GiB  fits
#  4-bit weights, 16-bit cache, batch 16 at  8192 ctx: weights   3.7 + cache  16.0 =  19.7 GiB  fits
#  4-bit weights,  8-bit cache, batch 16 at  8192 ctx: weights   3.7 + cache   8.0 =  11.7 GiB  fits
#  4-bit weights,  8-bit cache, batch 32 at  8192 ctx: weights   3.7 + cache  16.0 =  19.7 GiB  fits

The last two rows are the ones worth remembering. Halving the cache precision buys more headroom here than halving the weight precision did, because at batch 16 and 8k tokens the cache is larger than the weights. And the room you free goes straight into concurrency: the same 19.7 GiB serves sixteen users with a 16-bit cache or thirty-two with an 8-bit one.

Quantize a matrix by hand#

Quantization in its simplest form is three operations: divide by a scale, round to an integer, clamp to the range the integer can hold. Everything the methods argue about is how the scale is chosen and how many weights share one. Start with a weight matrix shaped like a real one, which means mostly small values and a couple of channels with a much wider range.

import torch

torch.manual_seed(0)

W = torch.randn(512, 512) * 0.02
W[7] *= 25.0                                     # two loud output channels
W[200] *= 12.0

def quantize(W, bits, per_channel):
    """Symmetric integer quantization, returned dequantized so the error is visible."""
    qmax = 2 ** (bits - 1) - 1                   # 127 at int8, 7 at int4
    amax = W.abs().max(dim=1, keepdim=True).values if per_channel else W.abs().max()
    scale = amax / qmax
    q = torch.clamp(torch.round(W / scale), -qmax - 1, qmax)
    return q * scale

def rel_err(W, Wq):
    return float((Wq - W).pow(2).mean().sqrt() / W.pow(2).mean().sqrt())

print("abs max %.4f   std %.4f   ratio %.0f"
      % (W.abs().max(), W.std(), W.abs().max() / W.std()))
Wq = quantize(W, 8, per_channel=False)
print("int8 per-tensor: scale %.3e  rel RMS err %.4f"
      % (W.abs().max() / 127, rel_err(W, Wq)))

# abs max 1.6916   std 0.0313   ratio 54
# int8 per-tensor: scale 1.332e-02  rel RMS err 0.1229

That is the whole problem in one number. The scale has to be large enough that the biggest weight does not clip, so it is set by the single loudest value in the tensor. Every ordinary weight is then rounded to a grid whose spacing was chosen for somebody else.

Where the error actually lands#

row_err = (Wq - W).pow(2).mean(dim=1).sqrt() / W.pow(2).mean(dim=1).sqrt()
loud = W.abs() > 0.2
print("weights above 0.2: %d of %d (%.2f%%)"
      % (int(loud.sum()), W.numel(), 100 * float(loud.float().mean())))
print("rel err: median row %.4f   row 0 %.4f   row 7 (the loud one) %.4f"
      % (row_err.median(), row_err[0], row_err[7]))

# weights above 0.2: 559 of 262144 (0.21%)
# rel err: median row 0.1922   row 0 0.1928   row 7 (the loud one) 0.0079

This is the result worth carrying away, because it is the opposite of what most people assume. Outliers do not get damaged by quantization. They get served perfectly, at the expense of the 99.79 per cent of weights that are nothing like them. Once you see it that way, every method in the field reads as a different answer to the same question: how do we stop a handful of values from setting the grid for everyone else.

Per-channel, and what it buys#

The cheapest answer is to stop sharing. Give each output channel its own scale, so a loud row sets the grid only for itself. The code above already supports it; the only change is which axis the maximum is taken over.

print("bits  scheme       rel RMS err   max abs err")
for bits in (8, 4, 3):
    for name, pc in (("per-tensor", False), ("per-channel", True)):
        Q = quantize(W, bits, per_channel=pc)
        print("%4d  %-11s  %11.4f   %11.4f"
              % (bits, name, rel_err(W, Q), (Q - W).abs().max()))

# bits  scheme       rel RMS err   max abs err
#    8  per-tensor        0.1229        0.0067
#    8  per-channel       0.0076        0.0067
#    4  per-tensor        0.6537        0.1207
#    4  per-channel       0.1383        0.1206
#    3  per-tensor        0.7146        0.2818
#    3  per-channel       0.3243        0.2817

Two things in that table repay a second look. The max absolute error column barely moves, because it is still dominated by the loud channel in both schemes; only the average improves, which is why RMS error is the number to watch rather than the worst case. And the gap widens as bits fall: at eight bits per-channel is a refinement, at three bits it is the difference between a usable tensor and a broken one. Group-wise quantization, which every serious 4-bit method uses, is the same idea taken further: one scale per 64 or 128 weights within a row rather than one per row.

n = W.numel()
print("bits  scales  bytes    vs 16-bit")
for bits, scales in ((16, 0), (8, 0), (8, 512), (4, 0), (4, 512)):
    total = n * bits / 8 + scales * 2             # scales stored at 16-bit
    print("%4d  %6d  %7d  %8.2fx" % (bits, scales, total, n * 2 / total))

# bits  scales  bytes    vs 16-bit
#   16       0   524288      1.00x
#    8       0   262144      2.00x
#    8     512   263168      1.99x
#    4       0   131072      4.00x
#    4     512   132096      3.97x

Now do it to a trained model#

Error on a matrix is a proxy. What you actually care about is whether the model still works, and the only way to know is to quantize the weights and rerun the eval. A three-layer classifier on the digits dataset trains in a few seconds on a CPU and is enough to show the shape of the answer.

import copy
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
Xtr = torch.tensor(Xtr, dtype=torch.float32) / 16.0
Xte = torch.tensor(Xte, dtype=torch.float32) / 16.0
ytr, yte = torch.tensor(ytr), torch.tensor(yte)

torch.manual_seed(0)
model = torch.nn.Sequential(
    torch.nn.Linear(64, 256), torch.nn.ReLU(),
    torch.nn.Linear(256, 256), torch.nn.ReLU(),
    torch.nn.Linear(256, 10),
)
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for _ in range(200):
    opt.zero_grad()
    torch.nn.functional.cross_entropy(model(Xtr), ytr).backward()
    opt.step()

def accuracy(m):
    with torch.no_grad():
        return float((m(Xte).argmax(1) == yte).float().mean())

def quantized_copy(m, bits, per_channel):
    c = copy.deepcopy(m)
    with torch.no_grad():
        for layer in c:
            if isinstance(layer, torch.nn.Linear):
                layer.weight.copy_(quantize(layer.weight, bits, per_channel))
    return c

with torch.no_grad():
    spread = [float(l.weight.abs().max(1).values.max() / l.weight.abs().max(1).values.median())
              for l in model if isinstance(l, torch.nn.Linear)]
print("channel range spread per layer:", [round(s, 2) for s in spread])
print("fp32 accuracy %.4f" % accuracy(model))
print("bits  per-tensor  per-channel")
for bits in (8, 4, 3, 2):
    print("%4d  %10.4f  %11.4f"
          % (bits, accuracy(quantized_copy(model, bits, False)),
             accuracy(quantized_copy(model, bits, True))))

# channel range spread per layer: [1.59, 1.47, 1.19]
# fp32 accuracy 0.9759
# bits  per-tensor  per-channel
#    8      0.9759       0.9759
#    4      0.9685       0.9722
#    3      0.9667       0.9704
#    2      0.1944       0.4741

The first line explains why per-channel helps so little here: the loudest channel in each layer is only 1.2 to 1.6 times the median, against a ratio of 54 in the synthetic matrix. A small well-conditioned network has no outlier problem to fix. Large language models do, and the gap between the two experiments is exactly the reason quantizing a transformer is a research area and quantizing this MLP is a one-liner.

The other thing to take from the table is the shape of the curve. Quality does not degrade smoothly as bits fall; it is flat, then it slopes, then it falls off a cliff between three and two bits. That cliff is why the field settled on four bits as the working floor for weights, and why claims about two-bit models always come with a method attached rather than a rounding rule.

Common approaches#

• Weight-only quantization stores weights in 8 or 4 bits and computes in higher precision. It is the most common for serving LLMs, because generation is limited by reading weights from memory.
• GPTQ and AWQ are widely used methods for 4-bit weights that use a small calibration set to decide how to round with less damage. Both are doing a smarter version of what you did above: GPTQ rounds one column at a time and corrects the remaining columns for the error it just made, AWQ finds the channels the calibration activations care about and scales them so they survive.
• GGUF is the file format used by llama.cpp for running quantized models on laptops and CPUs. Its Q4_K_M and similar names encode a bit width plus a grouping scheme, not a separate algorithm.
• bitsandbytes loads Hugging Face models in 8 or 4 bits with one argument, convenient for experiments and for QLoRA fine-tuning.

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=BitsAndBytesConfig(load_in_4bit=True),
    device_map="auto",
)

Activation quantization is a separate and harder problem, and worth knowing the name of. Weights are fixed, so you can look at them all and pick a scale. Activations depend on the input, so the scale must either be calibrated on sample data and hoped to hold, or computed at runtime, and transformer activations carry outliers far more extreme than the weights do. That is why the common production answer is weight-only, and why the papers about activation outliers exist at all.

Measure what you lost#

Perplexity on held-out text is a quick check: a large rise means damage. It is not enough on its own, because it averages over everything and the damage is not evenly spread. Run your task eval on both versions, and look especially at long inputs, arithmetic, code and less common languages, which degrade first. The pattern from the matrix experiment repeats at the model level: the average looks fine while a specific capability has quietly gone.

You can already say why generating tokens is expensive. This section is about the techniques that make it less so: batching that never pauses to refill, a KV cache allocated in pages, a small model drafting tokens for a large one to check. Each has a workload it suits and a workload where it does nothing or makes things worse, so the section is built around predicting which will help before you spend a week implementing it.

Two workloads, one model#

A generation request has two phases that behave nothing like each other. Prefill reads the whole prompt at once: every token goes through every layer together, so each weight is read once and used for hundreds or thousands of positions. That is a large matrix multiplication and it is compute-bound. Decode produces one token at a time: each step reads every weight again and uses it for a single position, which is very little arithmetic for a great deal of memory traffic, and it is memory-bound.

Prefill    scales with prompt length      compute-bound     sets time to first token
Decode     scales with output length      memory-bound      sets time per output token

Almost every technique below is an answer to the second row. Quantization helps because there are fewer weight bytes to read. Batching helps because the same read serves more requests. Speculative decoding helps because a memory-bound step has spare arithmetic going unused.

Roofline reasoning#

Arithmetic intensity is the number of floating point operations performed per byte moved to and from memory. Every accelerator has its own ratio of peak compute to peak bandwidth. If your intensity is below the hardware's ratio you are memory-bound, and no faster arithmetic will help; above it you are compute-bound.

arithmetic intensity = FLOPs performed / bytes moved

card with 1000 TFLOP/s and 3 TB/s  ->  ratio about 330 FLOP per byte

decode at batch 1    about 2 FLOP per weight byte   ->  far below: memory-bound
decode at batch 256  about 500 FLOP per weight byte ->  above: compute-bound
prefill, 2k prompt   thousands of FLOP per byte     ->  compute-bound

The useful consequence is that you can compute the floor on decode speed before renting anything. At batch size one, a decode step must read every weight, so the time per token cannot be less than the weight bytes divided by the memory bandwidth.

8B parameters at 16-bit  = 16 GB read per token
card with 2 TB/s         = 16 / 2000 s = 8 ms per token = about 125 tokens/s

the same model at 4-bit  = 4 GB  -> 2 ms -> about 500 tokens/s (ceiling)

Continuous batching#

Static batching collects requests, runs them together and returns when the longest finishes. A request wanting 20 tokens waits behind one wanting 500, and the GPU spends the tail of the batch mostly idle. Continuous batching admits and retires sequences at every decode step instead, so a finished sequence leaves immediately and a waiting one takes its slot.

It wins so much because decode is memory-bound. One step reads the weights once whether it serves one sequence or sixty, so the extra sequences use arithmetic that was being wasted. Throughput rises nearly linearly with batch size until either the KV cache fills the remaining memory or the batch grows large enough to become compute-bound, which is where the roofline arithmetic tells you to stop.

Paged attention#

The naive way to hold a KV cache is to reserve, per sequence, enough contiguous memory for the maximum length it might reach. A request that stops after 200 tokens of a possible 8,000 has held onto memory nobody could use. Paged attention allocates the cache in small fixed-size blocks, the way an operating system pages memory, and hands them out as a sequence grows.

Waste falls from most of the reserved cache to under one block per sequence, and the blocks make prefix sharing possible: two requests with the same system prompt can point at the same blocks until they diverge. The work that introduced it reported two to four times the throughput at the same latency, which is concurrency bought purely by not wasting memory.

Speculative decoding#

A small draft model generates the next few tokens cheaply. The large model then checks all of them in a single forward pass, because verifying k proposed positions costs about what one decode step costs: the weights are read once either way, and a memory-bound step has the arithmetic to spare. Tokens are accepted up to the first one the large model would not have produced, and generation continues from there.

draft proposes   the cat sat on the mat
large verifies   the cat sat on the  |  rug      -> 5 accepted, 1 corrected

one large-model pass produced 6 tokens instead of 1

Whether it pays depends on the acceptance rate. Each round costs the draft model's k steps plus one large pass, so the draft has to be both much cheaper, commonly ten to twenty times, and right often enough. Acceptance in the region of two thirds and above is where the arithmetic usually works; below that the drafting costs more than the accepted tokens save. Acceptance is also task-dependent, which is why it should be measured on your own traffic rather than taken from a paper. Code and structured output accept well; open-ended prose accepts worse.

The one place it disappoints is a server already batching heavily. It spends spare compute, and at large batch sizes there is none spare, so it can make throughput worse while still improving latency for a single stream.

You can already say that prefill is compute-bound and decode is memory-bound, and that a KV cache is what makes the second one bearable. This section writes both loops over a one-layer model in NumPy, checks they agree token for token, and counts the operations each performs — because the interview question is usually not what a KV cache is but what exactly it saves, and the honest answer is arithmetic rather than adjectives.

A model small enough to fit on a page#

One embedding table, one causal attention layer, one projection back to vocabulary size. It is not a good language model and it is not meant to be; it is the smallest thing with the structure that makes caching work.

import numpy as np

rng = np.random.default_rng(0)
V, d = 64, 32                                      # vocabulary, model width

E = rng.normal(size=(V, d)) * 0.1                  # token embeddings
W_q, W_k, W_v, W_o = (rng.normal(size=(d, d)) * 0.1 for _ in range(4))
W_lm = rng.normal(size=(d, V)) * 0.1               # projection to logits

work = {"proj": 0, "scores": 0}                    # counters, to measure rather than claim


def softmax(z, axis=-1):
    z = z - z.max(axis=axis, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=axis, keepdims=True)

The naive loop#

The obvious way to generate is to append the new token to the sequence and run the model over the whole thing again. It is correct, it is what every first implementation does, and the wasted work is easy to miss because each individual pass looks reasonable.

def forward(tokens):
    x = E[tokens]                                  # (T, d)
    T = x.shape[0]
    q, k, v = x @ W_q, x @ W_k, x @ W_v            # (T, d) each
    work["proj"] += 3 * T

    scores = q @ k.T / np.sqrt(d)                  # (T, T)
    work["scores"] += T * T
    scores = np.where(np.triu(np.ones((T, T), dtype=bool), 1), -np.inf, scores)
    ctx = softmax(scores) @ v                      # (T, d)
    return (ctx @ W_o) @ W_lm                      # (T, V)


def generate_naive(prompt, n_new):
    tokens = list(prompt)
    for _ in range(n_new):
        logits = forward(tokens)[-1]               # every row computed, one row used
        tokens.append(int(np.argmax(logits)))
    return tokens

Two things are recomputed on every step. The keys and values for every token already in the sequence, which have not changed and cannot change — the causal mask guarantees that a token's key and value depend only on itself and what came before it, never on what comes after. And the full square score matrix, of which only the last row is ever read.

That prefix invariance is not an incidental property. It is the causal mask, restated, and it is the entire reason a cache is sound rather than an approximation. If you proved it as an assertion when you wrote attention — truncating the sequence leaves earlier outputs unchanged — this is what you were proving it for.

Prefill, then decode#

The cached version splits generation into two phases that run different code. Prefill processes the prompt in one batched pass, exactly as the naive forward does, and keeps the keys and values it computed. Decode then handles one token at a time, computing three vectors rather than three matrices, appending two of them to the cache and attending over everything stored.

def prefill(tokens):
    x = E[tokens]                                  # (T, d)
    T = x.shape[0]
    q, k, v = x @ W_q, x @ W_k, x @ W_v
    work["proj"] += 3 * T

    scores = q @ k.T / np.sqrt(d)                  # (T, T)
    work["scores"] += T * T
    scores = np.where(np.triu(np.ones((T, T), dtype=bool), 1), -np.inf, scores)
    ctx = softmax(scores) @ v
    logits = ((ctx @ W_o) @ W_lm)[-1]              # (V,) - only the last position matters
    return {"k": k, "v": v}, logits                # keep k and v, discard q


def decode_step(token, cache):
    x = E[token]                                   # (d,) - one token, not a sequence
    q, k, v = x @ W_q, x @ W_k, x @ W_v            # (d,) each
    work["proj"] += 3

    cache["k"] = np.vstack([cache["k"], k])        # (t, d), one row longer
    cache["v"] = np.vstack([cache["v"], v])

    # no mask: everything in the cache is in the past, the future does not exist yet
    scores = cache["k"] @ q / np.sqrt(d)           # (t,) - a vector, not a matrix
    work["scores"] += scores.shape[0]
    ctx = softmax(scores) @ cache["v"]             # (d,)
    return (ctx @ W_o) @ W_lm                      # (V,)

def generate_cached(prompt, n_new):
    cache, logits = prefill(list(prompt))
    tokens = list(prompt)
    for _ in range(n_new):
        nxt = int(np.argmax(logits))
        tokens.append(nxt)
        logits = decode_step(nxt, cache)
    return tokens

The same output, and the work saved#

A cache is not an approximation, so the first thing to check is that the two loops produce the same tokens. Then count.

prompt, n_new = [5, 9, 2, 41], 60

work = {"proj": 0, "scores": 0}
a = generate_naive(prompt, n_new)
naive = dict(work)

work = {"proj": 0, "scores": 0}
b = generate_cached(prompt, n_new)
cached = dict(work)

print(a == b)                                      # True
print(naive)                                       # {'proj': 6030, 'scores': 85330}
print(cached)                                      # {'proj': 192,  'scores': 2086}

The shape of the saving matters more than these particular numbers. At step t the naive loop performs work proportional to t for the projections and to t squared for the scores, so over n steps it is quadratic and cubic respectively. The cached loop performs constant projection work per step and work proportional to t for the scores, so over n steps it is linear and quadratic. Every measurement is one power of n cheaper.

                    naive total        cached total
projections         O(n^2)             O(n)
score elements      O(n^3)             O(n^2)
memory held         O(n) tokens        O(n) cache rows

What the cache costs#

The saving is bought with memory, and in a real model that memory is the constraint that decides how many requests a GPU can serve at once. Two vectors per token, per layer, per attention head, at whatever precision the cache is kept in.

bytes = 2 * layers * kv_heads * head_dim * bytes_per_number * tokens * batch

# a 70B-class model: 80 layers, 8 KV heads (grouped-query), head_dim 128, fp16
# 2 * 80 * 8 * 128 * 2 = 327,680 bytes per token, about 0.31 MB

# 8,000 tokens of context      ->  about 2.5 GB per sequence
# 32 concurrent sequences      ->  about 80 GB, the whole card

Two consequences follow that get asked about. The cache grows with the number of tokens, not with how fast they arrive, so a long conversation costs memory for its whole lifetime even while it is idle. And because a prefix produces the same keys and values every time, two requests sharing a system prompt can share those rows — which is what prompt caching sells, and what paged attention makes possible at block granularity.