Asked about Memory arithmetic and quantizing a model yourself
Will an 8 billion parameter model serve 16 concurrent users with 8k contexts on a 24 GB GPU?
An answer that works
Work it out. At 16-bit the weights alone are 16 GB. The KV cache for a typical 8B design is about 128 KiB per token, so 16 users at 8k tokens need about 16 GiB. That does not fit. With 4-bit weights, about 4 to 5 GB, and a quantized or smaller cache, or fewer concurrent long contexts, it becomes feasible.
Then they push further
What would you change first?
Quantize the weights to 4 or 8 bits and measure quality. Then look at the cache: cap context length, quantize the cache if the serving engine supports it, and use an engine that allocates cache memory in pages so it is not wasted.
How do you know the 4-bit model is good enough?
Run the task eval on both versions and compare, paying attention to the categories that degrade first, such as long inputs and arithmetic. Perplexity alone is not enough.
What gets marked up
Strong answers do the arithmetic for weights and KV cache out loud. Forgetting the cache is the common miss.
Reading an answer is not being able to give one. The section that teaches this is Memory arithmetic and quantizing a model yourself, 4 hours of Quantization & inference optimisation.
Learn this properlyAlso asked about Quantization & inference optimisation