Reading the results section critically
By the end: You can tell a strong empirical result from a weak one by looking at the tables.
- What a results table is and is not saying
- Error bars and what their absence means
- Benchmark choice as a signal
- Recognising a result that will not replicate
New to reading papers? Research is published as papers: eight or ten page documents, most of them free on arXiv, each reporting one result. Reading one is not reading it front to back. It is a triage — the abstract and the figures to find what is being claimed, the method section to see what was actually done, the results table and the baseline it was compared against to judge whether the claim survives, and the full text only if the paper has earned it. Why it matters
The results section is where a paper is most carefully written and where you should read most slowly. Nobody fabricates numbers; what happens instead is that the honest ones get arranged so the best reading is the obvious one. This section is about the specific things to check — whether a gap is larger than its uncertainty, what a missing error bar means, why the choice of benchmark is itself a claim, and how to recognise a result that will not survive contact with your own data.
Is the gap bigger than the noise?#
This one question settles most tables, and the arithmetic takes a minute. A benchmark score is a proportion measured on a finite number of questions, so it carries the uncertainty of any proportion: roughly the square root of p times one minus p over n. Here is a plausible leaderboard on a thousand-question benchmark.
from math import sqrt
n = 1000 # questions in the benchmark
for name, p in [("A", 0.742), ("B", 0.731), ("C", 0.728), ("D", 0.699)]:
se = sqrt(p * (1 - p) / n)
print(f"{name} {p*100:.1f}% 95% CI [{(p-1.96*se)*100:.1f}, {(p+1.96*se)*100:.1f}]")
# A 74.2% 95% CI [71.5, 76.9]
# B 73.1% 95% CI [70.4, 75.8]
# C 72.8% 95% CI [70.0, 75.6]
# D 69.9% 95% CI [67.1, 72.7]And the gap between the top two, taken directly: 1.1 points, with a standard error of 2.0 and an interval running from minus 2.8 to plus 5.0. It comfortably contains zero, and it contains B beating A by more than A beats B. To resolve a gap that size you would need roughly twelve thousand questions rather than one thousand, because the precision improves only with the square root of the sample.
The one caveat, in your favour
If both models were run on the same questions — and on a fixed benchmark they were — a paired comparison is tighter than this, because it removes the variation the models share. So treat these intervals as the conservative version. The point stands: a table that reports four scores to one decimal place and no uncertainty at all has not given you enough to rank them, and a paper that ran the paired test would have said so, because it would have helped.
Missing error bars#
An absent error bar is not a neutral omission. Producing one costs a few more runs, everyone knows it is expected, and the papers that report them are not a random sample. Read a bare number as a single run until told otherwise, and check the appendix before assuming the worst — it is often there, and it is often wider than the gap being claimed in the abstract.
what it says how to read it
"84.2" one run, probably
"84.2 +/- 0.3" over what? seeds, or test examples, or folds?
"84.2 +/- 0.3 (3 seeds)" good: now the gap can be judged
"84.2 (best of 5)" the maximum of 5 draws, which is biased upward
"84.2" in bold, competitor "84.0" bolding a difference of 0.2 is a claim, not a factBenchmark choice is a claim#
Which benchmarks appear is as informative as the numbers on them. A method evaluated on three datasets when the field's standard set has eight was probably evaluated on all eight. A new benchmark introduced in the same paper that proposes the method deserves particular care, because the method and the measure were designed together. None of this is misconduct — reporting your best result is human — but it changes what the table supports.
- Are the standard benchmarks for this problem all present, or a subset?
- Is the comparison against the strongest published baseline, or a convenient one?
- Are the baseline numbers copied from other papers, which may use different preprocessing, or rerun here?
- Does the improvement hold across every dataset, or come from one where the gain is large?
- Is the test set one the method's training data could plausibly have seen?
That last one has become the decisive question for anything involving a pretrained model. If a benchmark was published before the training data was collected, the questions and often the answers may simply be in the training set, and the score then measures memorisation. It is the first thing to check on a suspiciously strong language-model result, and the papers that have checked usually say so explicitly.
Results that will not replicate#
After a while these become recognisable on sight. The signals are consistent, and each has an innocent explanation that is sometimes even true.
signal what it often means
a large gain on one dataset, small the gain is that dataset, not the method
or negative elsewhere
improvement of the same size as the it has not been measured yet
seed-to-seed spread
a method with many components and the components are carrying nothing
no ablation
hyperparameters tuned for the method you are reading a tuning result
but stated as defaults for baselines
the gain shrinks as the baseline gets it was a baseline gap, not a method gain
stronger across the table
no code, and a detail-light method reproduction will find the missing details
sectionNone of this means distrusting papers. It means reading the results section as evidence with a known error bar rather than as a verdict, which is exactly what its authors would want from a careful reader — and it is what an interviewer is testing when they hand you a paper and ask what you think of it.
Where people go wrong
Trusting the abstract. It states what the authors hope the table shows, and the two diverge more often than you would expect.
If this comes up in an interview
What do you check first in a paper's results?
The absolute size of the improvement, whether there are error bars or multiple seeds, and whether the baselines were run under the same conditions. Then whether the claimed result holds across all settings or only some.
What makes you doubt a result will replicate?
Single-seed results, a large gain from a small change with no mechanism, an untuned baseline, no released code, and improvements that only appear on the authors' own benchmark.
Build this: Take three papers and, for each, write one sentence on what the results table actually supports. Compare it against the abstract.
Watch it explained
This is one section of Reading papers, 19 hours across 4 sections. Read it in full to get the whole topic in order.