Asked about Why leaderboard numbers mislead
Why can benchmark scores be misleading?
An answer that works
Contamination, since public benchmarks end up in training data so a high score can be memorisation. Saturation, where remaining differences above about 90 percent are noise and label error. And construct validity, where the benchmark may not measure what its name claims.
Reading an answer is not being able to give one. The section that teaches this is Why leaderboard numbers mislead, 4 hours of Benchmarks & contamination.
Learn this properlyAlso asked about Benchmarks & contamination
Two papers report accuracy on the same benchmark. What has to be true before you can compare the numbers?How would you evaluate models for your own use case?A model tops a leaderboard. Should you use it?A new model tops a public leaderboard by two points. Would you switch to it?What makes a benchmark result trustworthy?