Asked about Building and reporting a benchmark you can trust
What makes a benchmark result trustworthy?
An answer that works
Items unlikely to be in training data, a setup identical across models, several prompts and seeds with the spread reported, confidence intervals, and a task that actually measures the capability claimed.
Reading an answer is not being able to give one. The section that teaches this is Building and reporting a benchmark you can trust, 4 hours of Benchmarks & contamination.
Learn this properlyAlso asked about Benchmarks & contamination
Two papers report accuracy on the same benchmark. What has to be true before you can compare the numbers?Why can benchmark scores be misleading?How would you evaluate models for your own use case?A model tops a leaderboard. Should you use it?A new model tops a public leaderboard by two points. Would you switch to it?