Asked about Building and reporting a benchmark you can trust
A new model tops a public leaderboard by two points. Would you switch to it?
An answer that works
Not on that basis. Two points may be within noise, the benchmark may be contaminated or saturated, and it may not measure my task. I would run it on my own eval set with the same prompts I use, and compare cost and latency too.
Then they push further
How would you check whether the model has memorised a benchmark?
Prompt it with the beginning of test items and see whether it reproduces the rest exactly, and compare accuracy on the originals against reworded versions. A large gap suggests memorisation.
What gets marked up
Strong answers name noise, contamination, saturation and construct validity, and fall back on a private eval.
Reading an answer is not being able to give one. The section that teaches this is Building and reporting a benchmark you can trust, 4 hours of Benchmarks & contamination.
Learn this properlyAlso asked about Benchmarks & contamination