Reading papers without reading papers

More is published each week than anyone can read, so the skill is not reading papers but deciding which ones to read, and how far. This section covers a three-pass method that gets you the contribution of most papers in about fifteen minutes, the claims worth being sceptical about, and a way of keeping up that does not eat every evening. Reproducing a paper is the deepest read available, and worth doing occasionally.

Most papers deserve fifteen minutes#

The machine-learning categories on arXiv take several hundred new submissions on a working day, and the share of those that will change anything you do is small enough that reading them properly is not a plan. Triage is the response: a fixed shallow look at many, a longer look at a few, and the full read reserved for the handful you will build on.

The part that takes discipline is that the shallow read has to be deliberate. Opening a paper, reading the introduction, losing interest on page four and closing the tab is also a shallow read, and it leaves you holding the motivation rather than the result. The three passes below differ from that mainly in having a defined output: each one ends in a written note and a decision about what to do next.

Three passes#

1. 1.Five minutes: title, abstract, figures, conclusion. What do they claim and does it matter to you? Most papers stop here.
2. 2.Thirty minutes: method and results. How did they do it, what did they compare against, how large is the effect?
3. 3.Several hours: full detail, and ideally reimplementation. Reserve this for papers you will build on.

Pass one, performed#

Here is a pass one done rather than described. The paper is a composite of the sort of thing that appears on a Tuesday: the title and the numbers are invented so the reasoning is visible, and no real work is being characterised. What matters is the note at the end, because the note is the only thing pass one produces.

PASS ONE  (5 minutes)
"Curriculum ordering improves instruction tuning for small language models"

CLAIM       sorting the supervised fine-tuning set easy-to-hard beats
            random order. Headline: +2.1 on a judged benchmark score.
SETUP       one model family (7B), one instruction set (52k), scored
            by a larger model acting as judge.
FIGURES     Fig 1 is a pipeline diagram and carries no information.
            Fig 2 is the result: two bars, no error bars.
CONCLUSION  claims the effect is general; future work is other sizes.

DOES IT TOUCH MY WORK?   yes - we shuffle our SFT set at random
COST IF TRUE             about a day: score and reorder an existing set
BIGGEST DOUBT            two bars and no error bars implies one run each
DECISION                 pass two. Cheap to act on if true, cheap to
                         check if not. Both halves of that matter.

The decision rule is not whether the paper is interesting. Almost all of them are interesting. It is whether the claim being true would change what you do, and what it would cost you to find out. A paper that would change nothing goes into the category of things you have heard of, which is a real and useful category to have, and you stop there without guilt.

The same five minutes on a paper you cannot check works the same way. The 2017 transformer paper claims an architecture built from attention alone, with the recurrence removed, reaching translation quality comparable to the recurrent systems of the time while training far faster because the computation parallelises. That is the whole of pass one on it: a claim, a setting, and the consequence if true. Everything anyone remembers about that paper is downstream of a sentence you can extract in five minutes.

Pass two, performed#

Thirty minutes, and now you read the experimental setup, the tables, and the paragraph underneath each table. You are not trying to understand the method well enough to implement it. You are answering one question — is the comparison fair — because everything else in the paper is downstream of the answer.

PASS TWO  (30 minutes, same paper)

METHOD      score all 52k instructions for difficulty with a small
            judge model, train in that order, same number of steps
            as the baseline.
BASELINE    random order, same steps, same hyperparameters. Fair on
            budget - which is the usual failure, and is not this one.
NUMBERS     random 6.31, curriculum 6.52. Delta +0.21. The abstract's
            "+2.1" is the same delta on a rescaled axis.
SEEDS       three seeds for the method. ONE for the baseline.
SPREAD      method seeds 6.44 / 6.52 / 6.61, so a spread of 0.17.
ABLATION    none on the difficulty score itself.
MISSING     no hard-to-easy arm. That is the control separating
            "ordering matters" from "this ordering matters", and it
            costs exactly one more training run.

VERDICT     the effect (0.21) is about the size of the method's own
            seed spread (0.17), and the baseline's spread is unknown
            because it was run once. Not established either way.
ACTION      not worth a pass three. Worth an afternoon of my own:
            reorder our set, three seeds each way, on our eval. That
            answers it for the only setup I care about.

That is the ordinary result of a pass two and it is worth sitting with. Nothing in the paper is dishonest and no number is fabricated. But the effect is the same size as the noise the paper itself discloses, the baseline was run once so its noise is unmeasured, and the one extra run that would distinguish the two explanations was not done. None of that was visible from the abstract. All of it was visible in half an hour, and the half hour converted an interesting claim into a specific experiment you can run on your own data.

Pass three is reimplementation#

The third pass is not slower reading. It is rebuilding the result, or one figure of it, from the description. Reserve it for work you will actually build on, which for most people is a few papers a year rather than a few a month, and scope it to a single figure rather than to the paper.

What it teaches is mostly the size of the gap between a paper and a working system. The things that are reliably missing, and that you will have to guess:

• The learning rate schedule, warmup length and optimiser epsilon, of which typically one of the three is stated.
• The exact prompt template, when the result depends on a model being prompted.
• Which version of the evaluation harness, which for judged benchmarks can move a score by more than the paper's effect.
• Whether numbers are the best checkpoint, the last checkpoint, or an average, and what the selection was made on.
• The preprocessing: deduplication, truncation length, how examples that exceed it were handled.
• The failed variants, which are never in the paper and are half of what the authors know.

Having guessed at that list once, you read differently afterwards and permanently. A sentence like 'we follow standard practice for training' stops reading as background and starts reading as three unreported decisions.

Where to be sceptical, and the sentence that triggers it#

Scepticism is not a mood, it is a small set of phrases that should make you stop and look something up. Most of them are not attempts to mislead; they are the ordinary compressions of a paper written to a page limit by people who know what they left out.

YOU READ                          IT OFTEN MEANS        WHAT TO CHECK

"we use the default               the baseline was      whether the method got a
 settings for the baseline"       not tuned             sweep the baseline did not

"results are averaged             the count is          how many seeds, and the
 over multiple runs"              unstated              spread, not just the mean

"we report the best               selection may have    whether selection used a
 checkpoint"                      used the test set     validation split at all

"due to space we omit             the component did     whether they claim credit
 the full ablation"               not help              for that component anyway

"our method is                    it lost               the table row where it
 competitive with X"                                    lost, and by how much

"qualitative examples             these are the best    how many were generated
 are shown in Figure 4"           of many               to obtain four

"we evaluate on a new             standard benchmarks   whether any established
 benchmark we introduce"          did not show it       benchmark appears at all

"the improvement is               it is small           the absolute numbers, and
 substantial"                                           the seed spread beside them

Two more that are structural rather than verbal. First, an effect reported only as a percentage improvement hides its own size: a 40% reduction in error can be 0.5 points, and you cannot tell without the absolute numbers, so find them. Second, the baseline is where most disputes actually live — a comparison against an undertuned baseline measures your tuning, and it is the first thing a reviewer or an interviewer will press on.

The inverse signal is worth naming too, because it is rare and it should raise your estimate of a paper sharply. A paper that reports a variant of its own method that did not work, or a setting where the effect disappears, is telling you the authors looked for the failure themselves. That is much more informative about whether the result will survive than any of its headline numbers.

Keeping up, with actual numbers#

Trying to read everything fails, and failing at it quietly costs more than not trying, because it comes with a permanent sense of being behind. A budget works better than an intention. Here is one that survives a full-time job, written out so that you can argue with the numbers rather than with the idea.

NINETY MINUTES, ONCE A WEEK, THE SAME SLOT EVERY WEEK

  40 titles    scanned from two category feeds, filtered to the
               four or five terms that touch my work          ~15 min
   8 abstracts read properly, drawn from those 40             ~20 min
   2 pass ones written out, drawn from those 8                ~15 min
   1 pass two at most, and often none                         ~30 min
     the rest  filing the notes where I will find them again  ~10 min

  PER MONTH    one pass two that turns into an experiment of my own
  PER QUARTER  one pass three: a single figure from a single paper,
               reimplemented until the number comes out or does not

  NOT IN THE BUDGET: reading anything the day it appears, unless it
  is directly in the way of what I am building this week.

Where the forty titles come from decides the quality of the whole exercise, and it matters more than the number. In descending order of value:

1. 1.Five to ten researchers whose judgement you trust, watched for what they say is worth reading rather than for what they publish. This is the highest-yield source by a wide margin, and it is the one that takes the longest to assemble.
2. 2.One or two curated digests, chosen because a named person selects for them rather than an engagement metric.
3. 3.The arXiv listing for one or two categories, filtered to a short keyword list. Scan titles only. Ten minutes, not thirty.
4. 4.One conference proceedings a year, read as an index of what the field decided was good, not as a reading list.
5. 5.Your own reading: papers that keep appearing in the related-work sections of things you already read are being pointed at by people who have read more than you.

The failure mode to watch for is a feed with no budget attached. Saving papers is not reading them, and a folder of 300 unread PDFs is a record of intentions rather than a library. If a paper has sat unread for a month, it lost to everything else you did that month, and deleting it costs you nothing you had.

Reproduce something#

Reimplementing one result teaches more than reading twenty papers, and the reason is specific: reading tells you what someone concluded, while reproducing tells you how much had to be decided that nobody wrote down. Scope it ruthlessly. Not the paper — one figure, at the smallest scale where the claimed effect should still appear, with your own baseline run first so you have a number to compare against before you have anything to defend.

Two outcomes, both worth the weekend. Either the number comes out, in which case you now own a working implementation of something and you know the six details the paper omitted; or it does not, and you have to decide whether the gap is your bug or the paper's. Resolving that question honestly is the same skill the rest of this topic is about, and it is far easier to practise on somebody else's claim than on your own.