Product judgement for AI interview questions
6 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Designing for a component that is sometimes wrong
Read the section that teaches thisIf this comes up in an interview
How do you design a feature around a model that is sometimes wrong?
Assume mistakes rather than trying to eliminate them. Show sources so users can verify cheaply, make correction a single gesture, set expectations honestly in the copy, and match the level of autonomy to the cost of being wrong.
Why are AI failures harder for users than ordinary software failures?
Ordinary software fails visibly. An AI feature fails by producing something confident and well written that happens to be wrong, so the user has no signal. Design has to supply the signal that the system cannot.
How do you decide whether a model is accurate enough to ship?
By comparing time saved against time spent, not by picking a threshold. The cost has three parts: checking every answer, redoing the ones the check catches, and the incidents caused by the ones it misses. On a twelve-minute task with a two-and-a-half-minute check, a check that catches 85 per cent of errors needs about 80 per cent model accuracy to break even; improve the check to 99 per cent and the bar falls to about 38 per cent. So the first lever is the check, not the model.
Then they push further
When is the answer that you should not build it?
When the cost of an error that escapes is high enough that break-even accuracy goes above anything achievable, or when the user cannot tell the output is wrong, or when the action is irreversible. Then it is a draft behind an approval step, or nothing.
What gets marked up. Strong answers produce a number and treat verification as the variable they control. Promising to evaluate carefully is not an answer.
Would you show a confidence score to users?
Only if it is calibrated, meaning that things marked 80 percent are right about 80 percent of the time. An uncalibrated number is worse than none, because it looks like information and misleads.
Knowing whether the AI feature actually helps
Read the section that teaches thisIf this comes up in an interview
You launched an AI writing assistant. How do you know it is helping?
Look at behaviour, not just ratings: how much of each draft survives editing, how often users regenerate, accept rates, and whether the time to finish the task dropped. Tie that to an outcome agreed before launch, with guardrails on complaints and cost.
Then they push further
Thumbs-up rate is 85%. Good news?
Hard to say. Only a small, unrepresentative share of users rate at all. I would check it against behaviour such as edit distance and abandonment, and read a sample of unrated sessions.
Users accept drafts but edit 70% of the text. What does that mean?
The drafts are a starting point but not good enough to send. Read the edits: they show exactly what users wanted instead, and they are excellent data for improving the prompt or building an eval.
What gets marked up. Marked up for behavioural signals and outcome metrics over ratings, and for treating user edits as data.
How do you decide what to fully automate?
By how often it happens and how costly a mistake is. Frequent, low-cost tasks can be automated. Costly ones get a draft that a person approves. Rare, costly ones are usually best left to people.
These are the questions for one topic. The whole question bank covers every topic on the site.