Quantization or a smaller model

Good at

• Cutting memory roughly in half at 8-bit and to about a quarter at 4-bit, against fp16.
• Keeping the larger model's knowledge and reasoning, which the smaller one does not have.
• Speeding up single-stream decoding, which is bandwidth-bound and so scales with bytes moved.
• Being a deployment-time change: same checkpoint, no retraining.
• Fitting a model onto hardware you already have rather than hardware you must buy.

Costs you

• Quality does drop. 8-bit is usually negligible; 4-bit is small but real and task-dependent.
• Degradation is uneven — long-context, code and maths tasks tend to suffer first.
• Kernel support varies by format and hardware, so the theoretical speedup is not always available.
• The KV cache is not shrunk by weight quantization, and at long context it dominates memory.
• You must re-run your evals: the published perplexity delta will not tell you about your task.

Good at

• Genuinely lower compute, not just fewer bytes per weight — throughput improves as well as memory.
• Full-precision behaviour, with no quantization artefacts to diagnose.
• Broad tooling support, since no special kernels or formats are involved.
• Being a much better base for fine-tuning, which often closes the gap on a narrow task.
• Lower latency end to end, including prefill.

Costs you

• Less knowledge and weaker reasoning, and on hard instances this shows.
• Failures are qualitatively different, so prompts tuned on the big model need redoing.
• Long-context behaviour is often much weaker than the advertised window suggests.
• You may end up fine-tuning to recover, which is a project rather than a config change.