Deployment & serving interview questions
10 questions across 4 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Running an AI service
Read the section that teaches thisIf this comes up in an interview
What is different about deploying an AI service?
Requests take seconds not milliseconds, concurrency is governed by the provider's rate limit rather than your CPU, streaming means long-lived connections, cost varies per request so traffic spikes are budget events, and an upstream you do not control can be slow or down.
Your model provider is having an outage. What should your service do?
Retry briefly with backoff, then fall back to an alternative model, then serve cached results if reasonable, then return an honest message. Decide this before the outage, because during one you will not.
What would you monitor on an AI service?
Latency at p50 and p95 split by step, error rate broken down by type, cost per request and per hour with an alert, token usage including cached tokens to confirm caching works, queue depth, and upstream provider health.
Your prediction endpoint has a 150 ms p99 target and is missing it. Where do you start?
By writing the latency budget: every component on the path with its p50 and p99, measured rather than guessed. On a typical path with two feature lookups, the model forward pass is under ten per cent of the tail and the database calls are three quarters of it, so optimising the model buys almost nothing. I would also count sequential round trips before counting milliseconds, since merging two lookups into one batched call is usually the single largest win available.
Then they push further
Can you just add up the components' p99s to get the end-to-end p99?
No, and it is wrong in the safe direction. Components rarely peak together, so the sum is an upper bound. On a seven-component path with 308 ms of summed tails, simulating the components as lognormals fitted to their measured p50 and p99 gives an end-to-end p99 nearer 200 ms. Either simulate or measure end to end; never report the sum as the latency.
What gets marked up. Marked up for measuring before optimising, for knowing the sum of tails is not the tail of the sum, and for counting round trips.
Operating the service day to day
Read the section that teaches thisIf this comes up in an interview
How should prompt changes be released?
Like code: versioned, tested against the eval suite before release, and rolled out gradually with the ability to revert. A prompt edit can change behaviour as much as a code change.
An AI feature starts misbehaving in production. What do you do first?
Check whether the provider is having issues, then what recently changed, including prompt edits and model versions. Mitigate first by rolling back or switching to a fallback, then diagnose.
How long should a canary run before you trust it?
Long enough to collect the sample the metric needs, which is a calculation rather than a judgement. A five per cent canary on a fifteen-requests-a-second service needs about sixteen hours to detect a two-tenths-of-a-point move in a one per cent error rate, and about fifteen hours for a small conversion change. Half an hour is fine for crashes, timeouts and latency, because those show up on nearly every request; it is not evidence about a business metric.
Then they push further
The canary looks fine after ten minutes. Can you ramp to 50 per cent?
For the guardrails, yes: ten clean minutes says the thing does not fall over. But ramping while measuring means the sample is drawn from a moving population, so the business comparison has to restart at the new share. Decide the share first, hold it for the calculated duration, and treat the early period as a smoke test rather than the experiment.
What gets marked up. Marked up for treating duration as a sample size, and for separating what a short canary can and cannot tell you.
What breaks a model in production that no offline evaluation can catch?
Three things, and they need different defences. Feature skew, where serving computes a feature differently from training, fixed by one implementation and preprocessing saved inside the artefact. A stale feature store, where the values are plausible but older than the request, which costs whatever the fast-moving part of the feature was worth. And a dependency upgrade changing preprocessing, which offline evaluation cannot see because the evaluation runs on the new library too, so both sides move together and agree.
Then they push further
How do you defend against the dependency case specifically?
A golden fixture: a handful of representative rows committed to the repository, the preprocessing output hashed including the column names, and the build failing when the hash moves. Regenerate the hash only as a reviewed change, never by rerunning the generator to make a red build pass.
What gets marked up. Marked up for naming all three, and for structural defences rather than more testing of the same kind.
Capacity planning and load testing for an LLM service
Read the section that teaches thisIf this comes up in an interview
You expect 50 requests per second at peak, each taking about 8 seconds. What capacity do you need?
By Little's law about 400 concurrent requests. Every layer, from server workers and connection pools to the provider's concurrency, requests per minute and tokens per minute limits, must support that with headroom. Then I would load test to confirm and plan throttling behaviour.
Then they push further
The provider throttles you at peak. What does your service do?
Retry with exponential backoff and jitter, honouring retry-after, cap the internal queue, prioritise interactive requests, fall back to another model or provider if one is set up, and show users a clear message rather than hanging.
What gets marked up. Marked up for Little's law, token-based limits, and jittered backoff.
Serving open models yourself
Read the section that teaches thisIf this comes up in an interview
Why does an inference server like vLLM get much higher throughput than a simple generation loop?
Continuous batching keeps the GPU full by adding and removing sequences at every step, and paged attention allocates KV cache memory in blocks so many more sequences fit. Together they raise the number of concurrent requests and the tokens per second per GPU.
Then they push further
Throughput is great but users complain about slow first tokens at peak. What do you change?
Limit batch size or concurrent sequences per instance to protect latency, add instances and scale on queue depth, separate long-context requests, and consider a smaller or quantized model for interactive traffic.
What gets marked up. Marked up for continuous batching, paged KV cache, and treating latency and throughput as a trade to be set by a target.
These are the questions for one topic. The whole question bank covers every topic on the site.