LLM-as-judge interview questions
6 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Grading with a model
Read the section that teaches thisIf this comes up in an interview
How do you know your LLM judge is any good?
Grade a sample by hand, making sure it contains real failures, and measure on labels the judge was not tuned on how often it catches the failures and how often it accepts the passes. Report those numbers alongside the score. Without them the score is unanchored. Poor results almost always mean the rubric is ambiguous rather than the judge being weak.
Then they push further
Your judge agrees with you 90% of the time. Good enough?
It depends where the disagreements are. If only 5% of outputs are failures, a judge that passes everything agrees 95% of the time and is useless. Measure separately how many real failures it catches and how many real passes it accepts.
How many labels do you need?
Enough that both the tuning half and the test half contain a reasonable number of failures, which usually means a few dozen of each. If failures are rare, go looking for them deliberately.
What gets marked up. Strong answers measure the judge against human labels on held-out data and look at failures and passes separately. Quoting overall agreement alone is marked down.
What are the biases of an LLM judge?
Preference for length, preference for its own style, sensitivity to the order of compared options, and general leniency. Mitigate by ruling length out in the rubric, using a different judge model, running both orders, and giving concrete examples of failures.
When would you use pairwise comparison rather than scoring?
When comparing two versions of a system on open-ended quality, such as which reply is more helpful. Asking which is better is more reliable than asking for an absolute score, because the judge does not need a calibrated scale. For specific failure modes, a pass or fail check per output is simpler and tells you more.
Should the judge explain its reasoning?
Usually yes, briefly, before the verdict. A short critique first tends to make the verdict more accurate, and when the judge disagrees with you it shows whether the rubric or the judge is at fault. Reasoning written after the verdict is just justification. The cost is extra output tokens on every call.
Running judges on live traffic
Read the section that teaches thisIf this comes up in an interview
How would you monitor answer quality in production with an LLM judge?
Run validated pass or fail judges for the main failure modes on a random daily sample, plus more of the high-stakes traffic. Track the failure rate over time with an error bar, read the flagged traces, and pin and revalidate the judge model when it changes.
Then they push further
The failure rate jumps from 4% to 9% overnight. What do you check first?
Whether anything changed on either side: a deploy, a prompt edit, a provider model update, a shift in traffic such as a new customer, or a change to the judge itself. Then read the newly flagged traces to see whether the failures are real.
Why not judge every request?
Cost and latency, and it adds little. A sample of a few hundred a day estimates the rate well. Judging everything only makes sense where each individual verdict drives an action, such as holding a message for review.
What gets marked up. Strong answers combine sampling, validated judges, trend tracking and reading flagged traces, and consider the judge itself changing.
Your judge marks 80% of traffic as passing. Is the pass rate 80%?
Not exactly. The judge misses some failures and wrongly fails some passes. With its catch and accept rates measured on labelled data you can correct the estimate, and you should report the uncertainty that comes with those measurements.
These are the questions for one topic. The whole question bank covers every topic on the site.