Evaluation interview questions

What is an eval?

A fixed set of inputs with a defined way of scoring the output, run on every change, producing a number you can compare between versions. It is the equivalent of a test suite for a system whose output is not deterministic.

Why can you not just use unit tests?

Model output varies between runs and there is usually no single correct string. Exact-match assertions flake and get deleted. Evals score properties and use graded judgement rather than equality.

How would you start evaluating an existing feature with no evals?

Read real traces first and write down how it fails, so the eval measures actual problems. Then take twenty real inputs, including ones that have failed, write down what a good output looks like for each, and score the current system to get a baseline. That number is what every future change is measured against.

You join a team with an LLM feature and no evals. What do you do in the first week?

Get full traces logged if they are not already. Read about a hundred real ones, write a note on the first failure in each, group the notes into failure modes and count them. Fix the obvious bugs directly, then build a small eval per recurring failure mode, code checks where possible and a validated judge where not. Only then start changing prompts.

What gets marked up. Strong answers start from reading data, not from choosing a tool or a metric, and they mention counting failures to prioritise. Weak answers name an eval framework in the first sentence.

Why note only the first failure in a trace?

Later failures are often consequences of the first. If retrieval fetched the wrong document, the wrong answer that follows is not a separate problem. Counting both inflates the downstream failure and hides where the fix belongs.

Why binary judges rather than a one to five scale?

A binary question about one failure mode can be checked against human labels and has a clear meaning. A five point scale has no agreed meaning between adjacent points, drifts between runs and between graders, and nobody can act on an average of 3.4.

What gets marked up. Marked up if the candidate ties this to validation: you can measure how often a binary judge agrees with your labels on both passes and failures.

Where do eval cases come from?

Real traffic, not imagination. Common cases in realistic proportions, every production failure you have seen, edge cases, and cases where the right answer is to refuse or return nothing.

Why split an eval set into train and test?

Tuning against every case overfits the prompt to that set, so the score improves while real performance does not. A held-out split you never inspect gives a number that still predicts reality.

How big should an eval set be?

Twenty real cases is enough to start and already catches regressions. A few hundred gives tighter confidence. Beyond that, variety matters far more than volume.

You have a bug that affects about 6% of traffic and a fix for it. How do you show the fix works?

Not with the general eval set. At 6% prevalence, a hundred random cases contain about six that exercise the bug, and a paired test on that few detects a working fix less than a tenth of the time. I would pull twenty to forty cases from the reports of that specific bug and use those to show it is fixed, then use the general set to check nothing else regressed and to say how much it mattered overall. Two sets, two questions.

What gets marked up. Strong answers separate the question 'did this fix work' from 'how much does this matter', and use a different set for each. Weak answers propose adding a few cases to the existing set and re-running it.

How do you evaluate something subjective like summary quality?

Use deterministic checks for everything checkable first. For what remains, break 'quality' into specific failure modes you have seen, such as unsupported claims or missing the main point, and write a pass or fail judge for each. Validate each judge against your own labels and report how well it catches failures alongside the score.

What are the known biases of an LLM judge?

Preference for longer answers, preference for its own writing style, sensitivity to the order of compared options, and general leniency. Each has a mitigation: rule length out in the rubric, use a different judge model, run both orders, and include concrete examples of failures in the rubric.

Your judge scores 4.2 out of 5. How much do you trust that?

Not much, for two reasons. Without a measured comparison against human grading the number is unanchored. And an average on a five point scale does not say what is wrong. I would replace it with pass or fail judges for specific failure modes, validated against my own labels, so the result reads as 'unsupported claims in 12% of summaries' rather than 4.2.

Your judge agrees with your labels 92% of the time. Is it good enough to use?

That number on its own cannot answer it, because it depends entirely on how often failures occur. If 12% of outputs are bad, a judge that simply passes everything already agrees 88% of the time, so 92% is barely above doing nothing. I would report Cohen's kappa, which subtracts the agreement two graders reach by chance, along with the catch rate on real failures and the acceptance rate on real passes. A judge at 92% agreement can sit anywhere from a kappa of 0.2 to 0.8 depending on where the mistakes fall.

What gets marked up. Strong answers immediately ask what the base rate is, and reach for kappa or for separate catch and acceptance rates. Very strong ones bring up the sample size of the failures. Quoting raw agreement as sufficient is the answer this question is designed to catch.

You are comparing two prompt versions with a pairwise judge. What do you do about position bias?

Run every comparison twice with the answers in both orders, and average. The gap between the win rate in one order and the other is the size of the bias, and it is worth reporting as a property of the judge. In a realistic case a fifteen-point pull towards whichever answer is shown first is enough to reverse the verdict on a version that is genuinely better, so judging in one fixed order can send you the wrong way.

How do you stop evals becoming expensive to run?

Run a small fast subset on every push and the full set nightly or before release. Cache results for unchanged inputs, and report the cost of the eval run itself so the trade-off stays visible.

What do you report from an eval run besides the pass rate?

Cost and latency, because changes routinely trade quality against them, and a quality-only report hides that. Also which specific cases changed state, since that is what you actually investigate.

How does an eval set stay relevant as a product changes?

Sample from real traffic periodically and add every production failure as a case. Version the set, and re-baseline when it changes so scores stay comparable.

Where do you set the threshold that fails a build?

By measuring, not by taste. Run the suite ten times against an unchanged commit and see how far the score moves on its own; that spread is the floor. Then simulate the gate against no change and against a regression of the size you care about, and pick the threshold where false alarms are rare and real breaks are still caught. On a hundred-and-twenty case set, failing on any drop at all fires on about two runs in five of an unchanged system, which is how teams end up ignoring their own eval.

What gets marked up. Strong answers measure run-to-run variance before choosing anything and treat the threshold as a trade-off with numbers on both sides. Weak answers give a round number with no reasoning, or propose failing on any regression at all.

How would you evaluate a customer support agent that can issue refunds?

Build tasks in a sandboxed copy of the systems it touches, and grade the end state: the right order changed, nothing else changed, the refund amount is correct. Separately check tool calls in the trace and have a validated judge assess the reply. Run each task several times and report both per run success and the share of tasks that pass every time.

What gets marked up. Strong answers grade the resulting state rather than the transcript, mention repeated runs, and bring up the cost of each failure type. Weak answers describe only an LLM judge reading the conversation.

Why is final-answer accuracy not enough for an agent?

It does not tell you where failures start, it ignores side effects on other data, and it ignores cost and step count. Two runs can reach the same answer, one cleanly and one after ten wasted tool calls.

What does it mean to find the first failing step, and why bother?

Walk each failed trace and mark the earliest step that went wrong. Tallied across runs, it shows which kind of step fails most, such as filling tool arguments. That points directly at the fix, where 'the agent is unreliable' does not.

Your change moved the eval from 80% to 83% on 60 cases. Do you ship it?

Not on that evidence. The interval on 60 cases is roughly plus or minus ten points, so the difference is well inside noise. I would look at the cases where the versions disagree, check run-to-run variance by rerunning the baseline, and grow the set if the decision matters.

What gets marked up. Strong candidates immediately question sample size and variance, and know to compare on paired cases. Very strong ones mention rerunning the baseline. Weak answers accept a three point gain at face value.

Why is a paired comparison better than comparing two pass rates?

Most cases pass or fail for both versions, and those add noise without information. Looking only at disagreements removes that shared noise, so a real difference shows up with far fewer cases.