The levers, in order
By the end: You can cut cost substantially without trading away quality, and you know which changes do trade.
- Prompt caching, the free win
- Trimming what you send
- Model routing
- Batching, and when quality has to give
New to cost & latency engineering? Every model call has a price, charged per token sent and received, and a delay before the answer arrives. This topic is the work of lowering both without the answers getting worse: sending fewer tokens, not paying twice for the part of the prompt that never changes, sending the easy requests to a smaller model, and starting to show the answer while it is still being written. Why it matters
Now that you know where the money actually goes, this is the list of things to do about it, deliberately ordered. The early entries are free: they cut the bill without touching quality, and they are worth exhausting before you consider anything that trades. The later ones — sending less, routing easy work to a cheaper model, batching — do trade, and this section is about knowing which you are choosing.
Free wins first#
Work in order. The early levers cost nothing in quality, and it is worth exhausting them before making any trade.
1. Prompt caching#
Providers can cache the processed form of a stable prompt prefix, and cached input is far cheaper than fresh input. If you have a long system prompt, tool definitions or few-shot examples, this is usually the single largest saving available.
Stable, cacheable, put first:
system prompt
tool definitions
few-shot examples
Volatile, put last:
conversation history
the user's question
anything with a timestampOne byte breaks it
Caching is prefix-matched from the start. A timestamp, a request id, or a randomly ordered JSON key anywhere early invalidates everything after it. Check cached token counts in your usage data. If they are always zero, something in your prefix is moving.
2. Send less#
- Trim tool results before they enter the context.
- Summarise or drop old conversation turns.
- Retrieve five relevant chunks instead of fifty mediocre ones.
- Delete instructions the model already follows. Long prompts accumulate lines nobody has tested.
3. Route by difficulty#
Most workloads are mostly easy. Send the easy cases to a small model and escalate only when needed. The saving is large because the split is usually lopsided.
def answer(question):
result = small_model(question)
if result.confidence < 0.7 or result.needs_escalation:
return large_model(question)
return resultVerify with your eval set that the routed system still passes. Routing is only free if quality holds, and the small model will fail on a category you did not anticipate.
4. Batch#
Most providers offer a batch mode at roughly half price for work that can wait hours. Anything offline, such as nightly enrichment, backfills or eval runs, should use it.
Then, and only then, trade quality#
- Lower the reasoning effort or thinking budget where the model supports it.
- Shorten the maximum output length, having first asked for brevity in the prompt.
- Move to a smaller model across the board, with eval numbers to justify it.
Where people go wrong
Switching to a cheaper model as the first move. It is the change most likely to hurt quality and is usually unnecessary, because caching and trimming often get you most of the way with no trade at all.
If this comes up in an interview
Your feature costs $0.40 per conversation and needs to be $0.10. What do you do?
Measure first and find which step dominates. Then in order: prompt caching on the stable prefix, trim what is sent including history and tool results, route easy cases to a smaller model, batch anything that can wait. Only after those do you trade quality, and with eval numbers to show the cost of the trade.
Then they push further
Most of the cost turns out to be a 6,000-token system prompt sent every turn. Now what?
Prompt caching first, since that prefix is identical every time. Then shrink it: move rarely needed material into tools or retrieval so it is only loaded when relevant.
Would you switch to a cheaper model?
For some requests, possibly. Route easy requests to a smaller model and keep the larger one for hard ones, but only after the eval shows the smaller model is good enough on those requests. Otherwise you trade quality for cost without knowing it.
What gets marked up. Marked up for measuring where the tokens go before changing anything, and for checking quality with evals before downgrading the model.
How does prompt caching work and how is it broken?
The provider caches the processed form of a stable prompt prefix and charges much less to reuse it. It is prefix-matched, so any change early in the prompt invalidates the rest. A timestamp or request id in the system prompt silently disables it, and the tell is cached token counts sitting at zero.
Then they push further
How would you check that caching works in production?
Read the cached token counts the API returns with each response and track the hit rate over time. A sudden drop usually means someone put something variable near the top of the prompt.
What gets marked up. Marked up for knowing the cache matches an exact prefix, so anything variable, such as a timestamp, must come after the stable parts.
When is model routing a bad idea?
When you cannot measure which requests are hard, or when a failure is expensive. Routing adds a decision that can be wrong, so it needs an eval showing quality holds and a fallback path when the small model is not confident.
Build this: Take a feature and cut its cost in half using only the free levers, then prove with your eval set that quality is unchanged. Report the before and after for cost, latency and pass rate.
Watch it explained
This is one section of Cost & latency engineering, 5 hours across 2 sections. Read it in full to get the whole topic in order.
Stop 34 of 50 on the AI Engineer route
0 of 50 topics done · 0h behind you, 603h to go
What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.