Choosing a model interview questions
5 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Choosing and changing models
Read the section that teaches thisIf this comes up in an interview
How would you choose between two models?
Use benchmarks only to shortlist, then run my own eval on real tasks and compare pass rate, cost and p95 latency together. Keep the prompt identical for the first comparison, then tune per model, since the ranking can change once each gets a fair prompt.
Then they push further
The cheaper model scores 3 points lower on your eval. Which do you pick?
Check first that 3 points is not noise. Then look at which cases it fails and what those failures cost. If they are low stakes and the saving is large, route those requests to the cheaper model. If they include costly errors, the saving is not worth it.
What gets marked up. Marked up for deciding on your own eval, cost and latency rather than public benchmarks, and for looking at which cases differ, not just the totals.
A provider is deprecating the model you use. What do you do?
The identifier should already be in configuration in one place, so the change is one line. Then re-run the eval before switching, because a newer model is not automatically better on a specific task and its failure cases move.
Why not always use the most capable model?
Cost and latency, and often neither is justified. Most workloads are mostly easy, so routing the easy majority to a smaller model and escalating the rest usually holds quality at a fraction of the spend. That claim needs an eval behind it.
Changing models without breaking things
Read the section that teaches thisIf this comes up in an interview
Your provider is retiring the model you use in three months. What is your plan?
Pick candidates, run the eval set on each, adapt the prompt for the best one, shadow real traffic, then roll out gradually with the old model still available. Start early, because prompt changes and review take longer than expected.
Then they push further
The new model scores the same on your eval but support tickets rise after launch. What happened?
The eval did not cover what changed. Common gaps are tone, length, refusals, or rare input types. Read the tickets and the traces behind them, add those cases to the eval, and roll back if the problem is serious.
What gets marked up. Marked up for eval comparison, prompt adaptation, shadow traffic and a staged rollout with rollback. Swapping the model name and shipping is marked down.
Why pin model versions?
So behaviour does not change without a deliberate decision. An alias that moves to a new version can alter output format or quality overnight, and you learn about it from users.
These are the questions for one topic. The whole question bank covers every topic on the site.