Serving open models yourself

Running an open model on your own GPU means running an inference server, such as vLLM or SGLang, which loads the weights and exposes an API your existing client code can talk to almost unchanged. This section covers what those servers do to go fast, which is batching requests continuously and allocating cache memory in pages, how to work out what will fit on a given card, and the trade between total throughput and how quickly one user sees their first token. Without that last part you can tune a server to excellent benchmark numbers and a bad experience.

One command#

pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192
# an OpenAI-compatible API on port 8000

Inference engines such as vLLM and SGLang handle batching, memory management and streaming, and expose an API compatible with common client libraries, so application code barely changes when you switch from a hosted provider.

Why they are fast#

• Continuous batching: new requests join the running batch at each generation step instead of waiting for the batch to finish, which keeps the GPU busy.
• Paged KV cache, the trick vLLM calls PagedAttention: cache memory is allocated in small blocks as needed rather than reserved for the maximum length, so many more sequences fit at once.

Sizing#

Weights take parameters times bytes per parameter; a 7B model in 16-bit needs about 14 GB. Everything left over is KV cache, which decides how many concurrent sequences of a given length fit — and that number, not the weights, is what determines whether the server can carry your traffic. The cache holds a key and a value vector per layer per token, so the cost per token is two, times the layers, times the number of key-value heads, times the head dimension, times two bytes.

GIB = 1024 ** 3
MODELS = [                     # layers, kv heads, head dim, params
    ("Qwen2.5-7B",    28, 4, 128,  7.6e9),
    ("Llama-3.1-8B",  32, 8, 128,  8.0e9),
    ("Llama-3.1-70B", 80, 8, 128, 70.6e9),
]
for name, layers, kv_heads, head_dim, params in MODELS:
    per_tok = 2 * layers * kv_heads * head_dim * 2      # K and V, 2 bytes each
    weights = params * 2 / GIB                          # bf16
    print(f"{name:<14} weights {weights:6.1f} GiB   KV {per_tok / 1024:6.1f} KiB per token")
    for card in [24, 48, 80]:
        free = card * 0.92 - weights                    # 8% for activations and overhead
        if free <= 0:
            print(f"   {card:>3} GB card: does not fit")
            continue
        toks = free * GIB / per_tok
        print(f"   {card:>3} GB card: {free:5.1f} GiB free -> {toks:9,.0f} cached tokens"
              f" = {toks / 4096:5.0f} sequences at 4k, {toks / 32768:5.1f} at 32k")

Qwen2.5-7B     weights   14.2 GiB   KV   56.0 KiB per token
    24 GB card:   7.9 GiB free ->   148,372 cached tokens =    36 sequences at 4k,   4.5 at 32k
    48 GB card:  30.0 GiB free ->   561,810 cached tokens =   137 sequences at 4k,  17.1 at 32k
    80 GB card:  59.4 GiB free -> 1,113,061 cached tokens =   272 sequences at 4k,  34.0 at 32k
Llama-3.1-8B   weights   14.9 GiB   KV  128.0 KiB per token
    24 GB card:   7.2 GiB free ->    58,809 cached tokens =    14 sequences at 4k,   1.8 at 32k
    48 GB card:  29.3 GiB free ->   239,688 cached tokens =    59 sequences at 4k,   7.3 at 32k
    80 GB card:  58.7 GiB free ->   480,861 cached tokens =   117 sequences at 4k,  14.7 at 32k
Llama-3.1-70B  weights  131.5 GiB   KV  320.0 KiB per token
    24 GB card: does not fit
    48 GB card: does not fit
    80 GB card: does not fit

Three readings, and the second one is the surprise. First, the weights are not the constraint on a 24 GB card serving a 7B model — they take 14 GB and leave 8, and it is the 8 that decides whether you can serve fourteen users or thirty-six. Second, Qwen2.5-7B and Llama-3.1-8B are nearly the same size and their cache costs differ by a factor of 2.3, because Qwen uses four key-value heads and Llama uses eight. On the same 24 GB card that is 36 concurrent 4k sequences against 14. Grouped-query attention is a serving decision as much as an architecture one, and it is not visible in the parameter count anybody quotes.

Third, look at the 32k column. Concurrency at long context collapses: the 24 GB card carrying 36 short sequences carries four long ones. This is why a service that is comfortable on short prompts falls over the week someone ships a feature that pastes a document in, and why long-context and interactive traffic usually want separate pools rather than one queue. Treat these figures as an upper bound — the engine reserves memory for its own scheduling and fragments its pages — and then confirm with a load test.

The trade-off#

Bigger batches mean more tokens per second in total and more cost-efficient serving, but each user's tokens arrive a little slower and time to first token rises under load. The mechanism is worth being precise about, because it decides which knob to turn. Generating a token reads every weight in the model, so a batch of sixteen reads them once and does sixteen times the arithmetic: throughput rises nearly linearly with batch size while the cost per token falls. What rises with it is the time each individual user waits, because their token is now produced in a step that is doing sixteen times the work, and because a new request may wait for a scheduling slot before its prompt is even processed.

So the two numbers pull against each other and there is no setting that optimises both. Decide the latency target first — a time to first token a person will accept, usually somewhere under a second for interactive use — then find the highest concurrency that still meets it, and add instances beyond that rather than batch size. Reporting tokens per second without the latency it was measured at is the most common way these benchmarks mislead.

Scaling#

A new GPU instance must download and load the weights before serving, so a cold start can take minutes. Scale on queue depth or pending requests rather than CPU, keep a minimum number of warm instances, and cache model weights on fast local disk or in the image registry close to the machines. Scaling on CPU utilisation is the specific error: the CPU on an inference box is nearly idle by design, so the autoscaler sees a quiet machine while the queue is thirty deep.

What to skip#

• Skip self-hosting entirely until you have a reason a hosted API cannot meet: data that may not leave, a model nobody hosts, or a volume at which the arithmetic genuinely favours a rented card. An idle GPU costs the same as a busy one.
• Skip writing your own batching loop. Continuous batching and paged attention are the two things the engines do well and the two things that are hard to reproduce.
• Skip the tokens-per-second benchmark that was measured at a concurrency nobody will run at. Quote throughput at the latency you promised.
• Worth memorising: the KV formula, that grouped-query attention changes concurrency more than model size does, and that cold starts are minutes rather than seconds.