Benchmarks & contamination interview questions
6 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Why leaderboard numbers mislead
Read the section that teaches thisIf this comes up in an interview
Two papers report accuracy on the same benchmark. What has to be true before you can compare the numbers?
The same items, the same prompt and number of examples, the same sampling settings, and the same rule for extracting and grading an answer. The last one is the one that gets skipped and it moves scores the most: on a set of replies that all give the correct letter, exact string match can score 12 per cent where taking the last option mentioned scores 100.
Then they push further
How would you check that in your own harness?
Run it twice with two grading rules and compare. If the gap between the graders is larger than the gap between the models, the harness is what you measured.
What gets marked up. Naming the answer-extraction rule is what separates a strong answer here. Most candidates stop at the prompt.
Why can benchmark scores be misleading?
Contamination, since public benchmarks end up in training data so a high score can be memorisation. Saturation, where remaining differences above about 90 percent are noise and label error. And construct validity, where the benchmark may not measure what its name claims.
How would you evaluate models for your own use case?
Shortlist with public benchmarks, then build a private set of real tasks, including production failures, and compare pass rate, cost and latency. A private set cannot be contaminated because it was never published.
A model tops a leaderboard. Should you use it?
It earns a place on the shortlist. Whether it is right for your task, at your cost and latency requirements, is a separate question that only your own eval answers.
Building and reporting a benchmark you can trust
Read the section that teaches thisIf this comes up in an interview
A new model tops a public leaderboard by two points. Would you switch to it?
Not on that basis. Two points may be within noise, the benchmark may be contaminated or saturated, and it may not measure my task. I would run it on my own eval set with the same prompts I use, and compare cost and latency too.
Then they push further
How would you check whether the model has memorised a benchmark?
Prompt it with the beginning of test items and see whether it reproduces the rest exactly, and compare accuracy on the originals against reworded versions. A large gap suggests memorisation.
What gets marked up. Strong answers name noise, contamination, saturation and construct validity, and fall back on a private eval.
What makes a benchmark result trustworthy?
Items unlikely to be in training data, a setup identical across models, several prompts and seeds with the spread reported, confidence intervals, and a task that actually measures the capability claimed.
These are the questions for one topic. The whole question bank covers every topic on the site.