Cost & latency engineering

Almost every attempt to make an AI feature cheaper starts by optimising the wrong thing. This section is about finding out first: instrumenting every call, and counting cost per completed task rather than per request, because a cheap call that has to be retried three times is not cheap. The same measurement answers the latency question, and the answer is usually somewhere nobody suspected.

The right unit#

Cost per API call is the wrong number. A cheaper model that needs three attempts and a retry is not cheaper. Measure cost per completed task, meaning the whole job done correctly, including retries and failed runs.

Model A:  $0.02 per call, succeeds 60% of the time
Model B:  $0.05 per call, succeeds 95% of the time

Cost per success:
  A:  0.02 / 0.60 = $0.033
  B:  0.05 / 0.95 = $0.053

A is cheaper, but only if a failure is free. If a failure costs a
retry, a support ticket or a lost customer, the maths changes.

Instrument everything#

Record tokens and cost for every call from the first day of a project. Adding it later means you cannot answer the first question anyone asks, which is where the money is going.

log.info("llm_call", extra={
    "task_id": task_id,
    "step": "extract",
    "model": model,
    "input_tokens": usage.input_tokens,
    "output_tokens": usage.output_tokens,
    "cached_tokens": usage.cache_read_input_tokens,
    "cost_usd": cost,
    "latency_ms": elapsed,
})

Where the cost usually is#

• A long system prompt sent on every call, uncached.
• Conversation history resent every turn, growing quadratically.
• Oversized tool results that stay in context for the rest of the run.
• A frontier model used for work a small one would do.
• Retries on failures that could have been avoided.

Latency is not the same problem#

Latency splits in two. Time to first token is the prefill pass over your prompt: it is compute-bound and it grows faster than the prompt does, because attention compares every token with every other one, so doubling a prompt more than doubles the wait before anything appears. After that, each output token costs roughly the same fixed amount, so the rest of the time is proportional to how much the model writes. A long prompt is not cheap in time; it is the most reliable thing to cut when the first token is slow.

Now that you know where the money actually goes, this is the list of things to do about it, deliberately ordered. The early entries are free: they cut the bill without touching quality, and they are worth exhausting before you consider anything that trades. The later ones — sending less, routing easy work to a cheaper model, batching — do trade, and this section is about knowing which you are choosing.

Free wins first#

Work in order. The early levers cost nothing in quality, and it is worth exhausting them before making any trade.

1. Prompt caching#

Providers can cache the processed form of a stable prompt prefix, and cached input is far cheaper than fresh input. If you have a long system prompt, tool definitions or few-shot examples, this is usually the single largest saving available.

Stable, cacheable, put first:
  system prompt
  tool definitions
  few-shot examples

Volatile, put last:
  conversation history
  the user's question
  anything with a timestamp

2. Send less#

• Trim tool results before they enter the context.
• Summarise or drop old conversation turns.
• Retrieve five relevant chunks instead of fifty mediocre ones.
• Delete instructions the model already follows. Long prompts accumulate lines nobody has tested.

3. Route by difficulty#

Most workloads are mostly easy. Send the easy cases to a small model and escalate only when needed. The saving is large because the split is usually lopsided.

def answer(question):
    result = small_model(question)

    if result.confidence < 0.7 or result.needs_escalation:
        return large_model(question)

    return result

Verify with your eval set that the routed system still passes. Routing is only free if quality holds, and the small model will fail on a category you did not anticipate.

4. Batch#

Most providers offer a batch mode at roughly half price for work that can wait hours. Anything offline, such as nightly enrichment, backfills or eval runs, should use it.

Then, and only then, trade quality#

• Lower the reasoning effort or thinking budget where the model supports it.
• Shorten the maximum output length, having first asked for brevity in the prompt.
• Move to a smaller model across the board, with eval numbers to justify it.