Asked about Why leaderboard numbers mislead
How would you evaluate models for your own use case?
An answer that works
Shortlist with public benchmarks, then build a private set of real tasks, including production failures, and compare pass rate, cost and latency. A private set cannot be contaminated because it was never published.
Reading an answer is not being able to give one. The section that teaches this is Why leaderboard numbers mislead, 4 hours of Benchmarks & contamination.
Learn this properlyAlso asked about Benchmarks & contamination
Two papers report accuracy on the same benchmark. What has to be true before you can compare the numbers?Why can benchmark scores be misleading?A model tops a leaderboard. Should you use it?A new model tops a public leaderboard by two points. Would you switch to it?What makes a benchmark result trustworthy?