Asked about Why leaderboard numbers mislead
A model tops a leaderboard. Should you use it?
An answer that works
It earns a place on the shortlist. Whether it is right for your task, at your cost and latency requirements, is a separate question that only your own eval answers.
Reading an answer is not being able to give one. The section that teaches this is Why leaderboard numbers mislead, 4 hours of Benchmarks & contamination.
Learn this properlyAlso asked about Benchmarks & contamination
Two papers report accuracy on the same benchmark. What has to be true before you can compare the numbers?Why can benchmark scores be misleading?How would you evaluate models for your own use case?A new model tops a public leaderboard by two points. Would you switch to it?What makes a benchmark result trustworthy?