Find out where the money goes
By the end: You can say exactly which part of your system is expensive before changing anything.
- Cost per completed task, not per call
- Instrumenting every call
- Finding the expensive path
- Latency and where it actually comes from
New to cost & latency engineering? Every model call has a price, charged per token sent and received, and a delay before the answer arrives. This topic is the work of lowering both without the answers getting worse: sending fewer tokens, not paying twice for the part of the prompt that never changes, sending the easy requests to a smaller model, and starting to show the answer while it is still being written. Why it matters
Almost every attempt to make an AI feature cheaper starts by optimising the wrong thing. This section is about finding out first: instrumenting every call, and counting cost per completed task rather than per request, because a cheap call that has to be retried three times is not cheap. The same measurement answers the latency question, and the answer is usually somewhere nobody suspected.
The right unit#
Cost per API call is the wrong number. A cheaper model that needs three attempts and a retry is not cheaper. Measure cost per completed task, meaning the whole job done correctly, including retries and failed runs.
Model A: $0.02 per call, succeeds 60% of the time
Model B: $0.05 per call, succeeds 95% of the time
Cost per success:
A: 0.02 / 0.60 = $0.033
B: 0.05 / 0.95 = $0.053
A is cheaper, but only if a failure is free. If a failure costs a
retry, a support ticket or a lost customer, the maths changes.Instrument everything#
Record tokens and cost for every call from the first day of a project. Adding it later means you cannot answer the first question anyone asks, which is where the money is going.
log.info("llm_call", extra={
"task_id": task_id,
"step": "extract",
"model": model,
"input_tokens": usage.input_tokens,
"output_tokens": usage.output_tokens,
"cached_tokens": usage.cache_read_input_tokens,
"cost_usd": cost,
"latency_ms": elapsed,
})Where the cost usually is#
- A long system prompt sent on every call, uncached.
- Conversation history resent every turn, growing quadratically.
- Oversized tool results that stay in context for the rest of the run.
- A frontier model used for work a small one would do.
- Retries on failures that could have been avoided.
Latency is not the same problem#
Latency splits in two. Time to first token is the prefill pass over your prompt: it is compute-bound and it grows faster than the prompt does, because attention compares every token with every other one, so doubling a prompt more than doubles the wait before anything appears. After that, each output token costs roughly the same fixed amount, so the rest of the time is proportional to how much the model writes. A long prompt is not cheap in time; it is the most reliable thing to cut when the first token is slow.
Streaming changes perception, not cost
Streaming shows tokens as they arrive, so the wait feels much shorter. Total time and total cost are unchanged. It is a real improvement to the experience and does nothing for the bill.
Where people go wrong
Optimising before measuring. Teams routinely spend a week shortening prompts when the actual cost was one oversized tool result being resent every turn.
If this comes up in an interview
How do you measure the cost of an AI feature?
Cost per completed task rather than per call, including retries and failed runs. Tag every call with a task id so a whole job can be summed, and record tokens, cost and latency for each.
What drives latency in an LLM call?
Two different things. Time to first token is prefill over the prompt, which is compute-bound and superlinear in prompt length because attention is quadratic, so a long prompt delays the first token more than proportionally. Everything after that is decode, roughly a fixed cost per output token, so it scales with how long the answer is. Shorten the prompt to fix a slow start; cap output tokens to fix a slow finish.
Does streaming make a response faster?
It makes it feel faster by showing tokens as they arrive. Total time to the complete answer and total cost are unchanged.
Build this: Add cost logging to an existing feature, run fifty real tasks, and produce a breakdown by step. Name the single most expensive step before you change anything.
This is one section of Cost & latency engineering, 5 hours across 2 sections. Read it in full to get the whole topic in order.
Stop 34 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.