Benchmark contamination

Also: test set contamination, data contamination

Benchmark questions having appeared in a model's training data, so its score measures recall of the answers rather than the ability being tested.

What people get wrong

It does not require anyone to have cheated. Public benchmarks get quoted, discussed and mirrored across the web, so a large enough crawl picks them up whatever the lab intended.

A definition is not an understanding. This is taught properly in Why leaderboard numbers mislead, part of Benchmarks & contamination.

Learn this properly