Eval-driven development interview questions
4 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Improving against a number
Read the section that teaches thisIf this comes up in an interview
How do you improve an AI system methodically?
Score a baseline, make one change, score again, keep or revert, repeat. Hold out a test split you never inspect during iteration and report that number, and track cost alongside quality since most improvements are trades.
Why hold out a test split when you are only tuning prompts?
Because prompt tuning overfits exactly as model training does. Iterating against every case drives the score up while real performance stays flat. The held-out number is the one that still predicts reality.
How do you know when to stop improving?
When several consecutive changes produce nothing, when the remaining failures are cases humans also disagree on, when further improvement costs more than it is worth, or when the test score has stopped moving while the train score keeps rising.
A week of improving one feature, worked through
Read the section that teaches thisIf this comes up in an interview
Tell me how you would improve an AI feature that 'feels unreliable'.
Read a hundred or so traces and categorise the failures, count them, and build an eval around the top modes with a held-out split. Then make one change at a time, starting with the most common failure, keep what improves the tune set at acceptable cost, and check the held-out set at the end.
Then they push further
Your first idea is to switch to a bigger model. Why might that be wrong?
If the failures come from outdated documents or lost context, a bigger model reads the same bad inputs. It also multiplies cost. Fix the causes the traces show first, then test a bigger model as one change among others.
How do you know you did not overfit to your eval?
Keep a held-out split you look at rarely, and check that it moves with the tuning set. Then watch production failure rates after release, since real traffic is the final held-out set.
What gets marked up. Strong answers start from data, change one thing at a time, weigh cost, and use a held-out set. This question often stands in for a behavioural round, so a real example with numbers scores highest.
These are the questions for one topic. The whole question bank covers every topic on the site.