Asked about The levers, in order
Your feature costs $0.40 per conversation and needs to be $0.10. What do you do?
An answer that works
Measure first and find which step dominates. Then in order: prompt caching on the stable prefix, trim what is sent including history and tool results, route easy cases to a smaller model, batch anything that can wait. Only after those do you trade quality, and with eval numbers to show the cost of the trade.
Then they push further
Most of the cost turns out to be a 6,000-token system prompt sent every turn. Now what?
Prompt caching first, since that prefix is identical every time. Then shrink it: move rarely needed material into tools or retrieval so it is only loaded when relevant.
Would you switch to a cheaper model?
For some requests, possibly. Route easy requests to a smaller model and keep the larger one for hard ones, but only after the eval shows the smaller model is good enough on those requests. Otherwise you trade quality for cost without knowing it.
What gets marked up
Marked up for measuring where the tokens go before changing anything, and for checking quality with evals before downgrading the model.
Reading an answer is not being able to give one. The section that teaches this is The levers, in order, 3 hours of Cost & latency engineering.
Learn this properly