Asked about Why leaderboard numbers mislead

Two papers report accuracy on the same benchmark. What has to be true before you can compare the numbers?

An answer that works

The same items, the same prompt and number of examples, the same sampling settings, and the same rule for extracting and grading an answer. The last one is the one that gets skipped and it moves scores the most: on a set of replies that all give the correct letter, exact string match can score 12 per cent where taking the last option mentioned scores 100.

Then they push further

How would you check that in your own harness?

Run it twice with two grading rules and compare. If the gap between the graders is larger than the gap between the models, the harness is what you measured.

What gets marked up

Naming the answer-extraction rule is what separates a strong answer here. Most candidates stop at the prompt.

Reading an answer is not being able to give one. The section that teaches this is Why leaderboard numbers mislead, 4 hours of Benchmarks & contamination.

Learn this properly

Also asked about Benchmarks & contamination

Why can benchmark scores be misleading?How would you evaluate models for your own use case?A model tops a leaderboard. Should you use it?A new model tops a public leaderboard by two points. Would you switch to it?What makes a benchmark result trustworthy?