Self-hosting or a provider
You have chosen an open-weights model. Do you run the serving stack, or rent it from an inference provider?
The short answer
Rent until a hard requirement or a large, provably steady bill says otherwise. Self-hosting is bought with engineer-months, and those are rarely in the spreadsheet that justified it.
Inference provider
Good at
- Per-token billing on open models, so idle time costs nothing.
- No cluster, no autoscaling, no on-call rota for GPU nodes.
- Switching between open models with a config change rather than a migration.
- Someone else keeping up with kernel and serving-stack improvements, which arrive monthly.
Costs you
- A margin on top of raw compute — you are paying for the operations you skipped.
- Data still leaves your network, so this solves cost and control but not residency.
- Limited control over quantization, batching and context length.
- Cold starts and queueing on less popular models.
Pick this when: You want open weights for cost or portability, not for data control.
Run it yourself
Good at
- Data never leaving your boundary — often the only reason this option exists.
- The lowest cost per token at high, steady utilisation, once the hardware is paid for.
- Tuning the whole stack: quantization, batching policy, speculative decoding, cache sizing.
- Predictable latency under load, because nobody else's traffic shares your GPUs.
- Serving your own fine-tunes and adapters without publishing them to anyone.
Costs you
- An inference service to operate, upgrade and be paged about.
- Capacity sized for peak, paid for at all hours.
- GPU supply is lumpy — you buy whole cards, so headroom is coarse and expensive.
- Real expertise in memory arithmetic, batching and the KV cache, or you will serve badly and blame the model.
Pick this when: A data-residency requirement, or sustained utilisation high enough that the margin you are paying exceeds the engineers you would spend.