Shipping and measuring interview questions
5 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Shipping and measuring
Read the section that teaches thisIf this comes up in an interview
Your eval score improved. Is the feature better?
It means the output moved in the direction the eval defines as better. Whether users are better off is a separate question that needs a live experiment with a metric chosen beforehand, plus guardrails on latency, cost and support load.
How would you roll out a change to an AI feature?
Behind a flag, internal first, then one percent watching errors and latency, then ten and fifty percent for a real comparison, then full. Instant rollback at every stage, and the success metric written down before launch.
Then they push further
What metric decides whether you keep it?
One agreed before launch that reflects user benefit, such as tasks resolved without escalation, plus guardrails like complaint rate, cost and latency that must not get worse.
What gets marked up. Strong answers include offline evals, a staged rollout behind a flag, a metric agreed in advance with guardrails, and a fast way to roll back.
What makes A/B testing an AI feature harder than a UI change?
Output varies per user so the treatment is not identical, quality effects are slow because trust builds and erodes gradually, cost is a genuine variable that belongs in the comparison, and novelty inflates early numbers.
The statistics of an online experiment
Read the section that teaches thisIf this comes up in an interview
You A/B test a new AI feature. After three days it shows a significant 4% lift. Ship it?
Not yet. Check whether the planned sample size was reached, since stopping at the first significant reading inflates false positives. Check the sample ratio, since a broken split invalidates everything. Look at guardrail metrics, and whether the effect is shrinking as novelty wears off.
Then they push further
The treatment group has 3% fewer users than control. Does that matter?
Yes, if the split was meant to be equal and the difference is statistically significant. It suggests a bug or users dropping out of one arm, often the arm with the new feature, which biases every metric. Find the cause before trusting the result.
How many users would you need to detect a 1 point lift on a 10% baseline?
Roughly 16 times 0.1 times 0.9 divided by 0.01 squared, about 14,400 per group, for 80% power at 5% significance.
What gets marked up. Strong answers raise peeking, sample ratio mismatch, guardrails and novelty. Doing the sample size estimate out loud stands out.
Why randomise by user rather than by request?
Users who see both versions blur the comparison and may change behaviour because of the inconsistency. Per-user assignment gives each person one consistent experience, which is what you want to measure.
These are the questions for one topic. The whole question bank covers every topic on the site.