Open weights or a hosted API
Do you run a model whose weights you have, or call someone else's endpoint?
The short answer
Build against an API, measure real volume and real quality needs, then price self-hosting against those numbers. The crossover is about utilisation, not principle.
Hosted API
Good at
- Starting today, with no capacity planning and no GPUs.
- Access to the strongest frontier models, which is where the hardest tasks are still won.
- Absorbing demand spikes without you owning the hardware for the peak.
- Getting improvements, longer contexts and new features without doing any work.
- Operational features — caching, batch discounts, structured output — that are real engineering to rebuild.
Costs you
- Per-token cost that scales linearly with success, with no floor to amortise against.
- Data leaves your network, which for some organisations ends the discussion.
- Rate limits and availability you do not control.
- Model versions get deprecated on someone else's schedule, and behaviour shifts under you.
- No access to logprobs, activations or weights, which rules out some techniques entirely.
Pick this when: Volume is uncertain, the task needs the strongest available model, or nobody on the team wants to be on call for a GPU.
Open weights, self-run
Good at
- Data never leaving your boundary.
- Cost that flattens with volume — at high steady utilisation, cost per token falls well below API pricing.
- Fine-tuning, quantizing, distilling, or reading the internals.
- A model that does not change unless you change it, which matters for regulated and reproducible work.
- Air-gapped and on-premises deployments that an API cannot serve at all.
Costs you
- You now run an inference service: capacity, batching, autoscaling, upgrades, on-call.
- Idle GPUs cost the same as busy ones, so low or spiky traffic is expensive per request.
- The best open models still trail the frontier on hard reasoning and long-horizon tasks.
- Long context is disproportionately expensive to serve yourself, because of the KV cache.
- Real engineering time, which is usually the largest line in the comparison and the one left out of it.
Pick this when: Steady high volume, a hard data-residency requirement, or you need to modify or inspect the model.
Where the field disagrees
Where the cost crossover actually sits. Self-hosting advocates quote cost per million tokens at full GPU utilisation, which almost no real workload sustains; API advocates quote list prices and omit that batch and caching discounts change them substantially. Both numbers are true of the workload the person quoting them has in mind.