Quantization & inference optimisation interview questions
14 questions across 4 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
What makes inference expensive
Read the section that teaches thisIf this comes up in an interview
What is the bottleneck in LLM inference?
Memory bandwidth. Generating each token requires reading all the model weights from memory, so the GPU is mostly waiting on memory rather than computing. That is why quantization speeds things up and why batching is nearly free.
Then they push further
Why is generating tokens slower per token than reading the prompt?
The prompt is processed in parallel, all tokens at once, which keeps the GPU busy computing. Generation makes one token at a time, and each step reads every model weight from memory for very little arithmetic, so it is limited by memory bandwidth.
How does that explain why batching helps?
Each generation step reads the weights once whether it serves one request or fifty, so adding requests uses compute that would otherwise sit idle, until memory for the KV cache runs out.
What gets marked up. Marked up for separating prompt processing from generation, naming memory bandwidth, and linking it to batching and quantization.
What does quantization trade?
Accuracy for memory and speed. 8 bit usually costs little, 4 bit costs something real and task-dependent. The loss is not uniform across tasks, so it has to be measured on your own eval rather than taken from a published aggregate.
Why is batching almost free?
The weights are read once and applied to every request in the batch, and reading them is the bottleneck. This is why hosted APIs are cheap and why self-hosting for low traffic is poor value.
When would you self-host a model?
When data cannot leave your infrastructure, when volume is high and steady enough to keep hardware busy, when you need a fine-tuned model nobody hosts, or when latency rules out a network round trip. Otherwise the arithmetic usually favours a provider.
Then they push further
How would you estimate whether it saves money?
Compare API cost at your real volume with GPU cost at the utilisation you would actually achieve, plus engineering time for serving, scaling and upgrades. Low or spiky traffic usually favours the API, because idle GPUs still cost money.
What gets marked up. Strong answers weigh data control, latency, steady high volume and customisation, and count the operating cost honestly.
Memory arithmetic and quantizing a model yourself
Read the section that teaches thisIf this comes up in an interview
Will an 8 billion parameter model serve 16 concurrent users with 8k contexts on a 24 GB GPU?
Work it out. At 16-bit the weights alone are 16 GB. The KV cache for a typical 8B design is about 128 KiB per token, so 16 users at 8k tokens need about 16 GiB. That does not fit. With 4-bit weights, about 4 to 5 GB, and a quantized or smaller cache, or fewer concurrent long contexts, it becomes feasible.
Then they push further
What would you change first?
Quantize the weights to 4 or 8 bits and measure quality. Then look at the cache: cap context length, quantize the cache if the serving engine supports it, and use an engine that allocates cache memory in pages so it is not wasted.
How do you know the 4-bit model is good enough?
Run the task eval on both versions and compare, paying attention to the categories that degrade first, such as long inputs and arithmetic. Perplexity alone is not enough.
What gets marked up. Strong answers do the arithmetic for weights and KV cache out loud. Forgetting the cache is the common miss.
Why does weight-only quantization speed up generation?
Generating each token requires reading all the weights from memory, and that reading is the bottleneck. Smaller weights mean less to read per token, so each step is faster even if the arithmetic is done at higher precision.
Making generation faster
Read the section that teaches thisIf this comes up in an interview
Why does prompt length change time to first token but barely change time per output token?
They are different workloads. Prefill processes the whole prompt in parallel, so it is compute-bound and scales with prompt length, and it is what time to first token measures. Each decode step reads the whole model to produce one token, so it is memory-bound and its cost is dominated by weight bytes over bandwidth, which does not change with prompt length. Only the growing KV cache adds a little.
Then they push further
Which one does your bill depend on?
Depends on the traffic. Long prompts with short answers, such as classification or extraction, are dominated by prefill and by input token pricing. Short prompts with long answers are dominated by decode, which is the slower and more expensive side per token.
How would you cut time to first token specifically?
Shorten the prompt, cache the shared prefix so it is not re-prefilled on every call, stream so the user sees output as it arrives, and keep long-context requests off the instances serving interactive traffic.
What gets marked up. Strong answers name the two phases, say which is compute-bound and which memory-bound, and then use that split to explain a serving decision.
How would you work out the fastest a model could possibly generate on a given GPU?
At batch size one, each token requires reading every weight, so divide the weight bytes by the memory bandwidth. An 8B model at 16-bit is 16 GB, and on a 2 TB/s card that is 8 ms per token, about 125 tokens per second, before the KV cache reads and any inefficiency. Quantizing to 4-bit cuts the bytes by four and raises the ceiling by roughly the same factor.
Then they push further
What changes at batch size 64?
The weight read is amortised across 64 sequences, so total throughput rises nearly linearly while each sequence's token rate holds up until the batch becomes compute-bound. What limits it first is usually KV cache memory, not arithmetic.
Why does continuous batching raise throughput so much?
Decode is memory-bound, so one step reads the weights once regardless of how many sequences are in the batch, and extra sequences use arithmetic that would otherwise be idle. Continuous batching keeps that batch full by admitting and retiring sequences at every step, rather than letting the whole batch wait for its longest member on mixed-length traffic.
What is speculative decoding, and what does it cost?
A small draft model proposes several tokens and the large model verifies them in one pass, accepting the longest prefix it agrees with. It is exact: the output distribution is the large model's. It costs the draft model's compute and only pays off above an acceptance rate of roughly two thirds, and it helps least on a server already running large batches, where there is no spare compute to spend.
A decode loop with and without a KV cache
Read the section that teaches thisIf this comes up in an interview
Write a generation loop with a KV cache, and say what it saves.
Prefill runs the prompt through in one batched pass, with the causal mask, and keeps the keys and values. Each decode step then embeds the single new token, computes one query, key and value vector, appends the key and value to the cache, attends over the whole cache with no mask, and produces logits for one position. It saves recomputing every past key and value — constant work per step instead of work proportional to the sequence so far — and it computes one row of attention scores instead of the full square matrix. The output is identical; a cache is exact, not an approximation.
Then they push further
Why is it exact?
Because of the causal mask. A token's key and value depend only on that token and the ones before it, never on what follows, so they cannot change as generation continues. That is the same property as a prefix of a sequence producing unchanged outputs when the rest is truncated.
Why is there no mask in the decode step?
The mask prevents attending to the future, and during decode the future has not been generated. Everything in the cache is already in the past. The mask is needed in prefill and in training, where many positions are processed together.
What does it cost you?
Memory, and it is usually the binding constraint on concurrency. Two vectors per token per layer per KV head. For an 80-layer model with 8 KV heads and head dimension 128 in fp16, about 0.31 MB per token, so 8,000 tokens is 2.5 GB for one sequence and thirty-odd concurrent sequences fill an 80 GB card. Grouped-query attention and paged allocation are both responses to that.
What gets marked up. Anyone can say the cache stores past keys and values. The follow-ups are where it separates: that it is exact and why, that decode needs no mask, and that the cost is memory rather than compute.
Why is the query not cached alongside the key and value?
Because it is used exactly once. The query for position t is compared against every key up to t to produce that position's output, and no later position ever refers to it — later positions bring their own queries. Keys and values, by contrast, are read again at every subsequent step, which is what makes them worth storing.
A naive loop regenerates 500 tokens from a 2,000-token prompt. How much work is wasted?
Each step re-projects the whole prefix, so step t does work proportional to t rather than to one. Summed over 500 steps from a base of 2,000 that is roughly 1.1 million token-projections instead of 500, a factor of about two thousand. The attention scores are worse still: the naive loop builds the full t-by-t matrix each step and reads one row of it, which is cubic in total against the cached version's quadratic.
How does prompt caching relate to the KV cache?
It is the same data, kept between requests rather than within one. A shared prefix produces identical keys and values every time it is processed, because nothing after it can affect them, so a server that has already prefilled that prefix can reuse those rows and skip the prefill work. That is why prompt caching is billed against input tokens and why it only helps when the prefix is genuinely identical from the first token.
These are the questions for one topic. The whole question bank covers every topic on the site.