Cost & latency engineering interview questions
6 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Find out where the money goes
Read the section that teaches thisIf this comes up in an interview
How do you measure the cost of an AI feature?
Cost per completed task rather than per call, including retries and failed runs. Tag every call with a task id so a whole job can be summed, and record tokens, cost and latency for each.
What drives latency in an LLM call?
Two different things. Time to first token is prefill over the prompt, which is compute-bound and superlinear in prompt length because attention is quadratic, so a long prompt delays the first token more than proportionally. Everything after that is decode, roughly a fixed cost per output token, so it scales with how long the answer is. Shorten the prompt to fix a slow start; cap output tokens to fix a slow finish.
Does streaming make a response faster?
It makes it feel faster by showing tokens as they arrive. Total time to the complete answer and total cost are unchanged.
The levers, in order
Read the section that teaches thisIf this comes up in an interview
Your feature costs $0.40 per conversation and needs to be $0.10. What do you do?
Measure first and find which step dominates. Then in order: prompt caching on the stable prefix, trim what is sent including history and tool results, route easy cases to a smaller model, batch anything that can wait. Only after those do you trade quality, and with eval numbers to show the cost of the trade.
Then they push further
Most of the cost turns out to be a 6,000-token system prompt sent every turn. Now what?
Prompt caching first, since that prefix is identical every time. Then shrink it: move rarely needed material into tools or retrieval so it is only loaded when relevant.
Would you switch to a cheaper model?
For some requests, possibly. Route easy requests to a smaller model and keep the larger one for hard ones, but only after the eval shows the smaller model is good enough on those requests. Otherwise you trade quality for cost without knowing it.
What gets marked up. Marked up for measuring where the tokens go before changing anything, and for checking quality with evals before downgrading the model.
How does prompt caching work and how is it broken?
The provider caches the processed form of a stable prompt prefix and charges much less to reuse it. It is prefix-matched, so any change early in the prompt invalidates the rest. A timestamp or request id in the system prompt silently disables it, and the tell is cached token counts sitting at zero.
Then they push further
How would you check that caching works in production?
Read the cached token counts the API returns with each response and track the hit rate over time. A sudden drop usually means someone put something variable near the top of the prompt.
What gets marked up. Marked up for knowing the cache matches an exact prefix, so anything variable, such as a timestamp, must come after the stable parts.
When is model routing a bad idea?
When you cannot measure which requests are hard, or when a failure is expensive. Routing adds a decision that can be wrong, so it needs an eval showing quality holds and a fallback path when the small model is not confident.
These are the questions for one topic. The whole question bank covers every topic on the site.