Reading in order to find problems

This is reading done for a different purpose from the reading you have been doing until now. Instead of extracting what a paper found, you go through it for what it did not do: the ablation that is absent, the setting the method was never tested in, the limitation admitted in one sentence and dropped. The section covers that habit, the written record it produces, and what to do with a result that surprises you, which in practice is where a disproportionate number of real directions start.

Read for what is missing#

Most people read a paper to learn what was found. To find problems you read for what was not done: the ablation that is absent, the setting the method was never tested in, the limitation stated in one sentence and never revisited.

Limitations sections are the most under-read part of any paper and the most useful. Authors are telling you, honestly, where the work stops. That boundary is where the next question lives, and it is stated by the people best placed to know, usually because a reviewer made them state it.

Here is the operation performed rather than recommended. The limitations paragraph below is the ordinary kind — constructed, but you have read a hundred of these — followed by what comes out of it when you read it as a source of work rather than as a disclaimer.

THE PARAGRAPH, AS WRITTEN

  "Our experiments are limited to English and to models of up to 7B
   parameters. We use a single retrieval corpus and do not vary the
   retriever. Finally, our evaluation is automatic; we leave human
   evaluation to future work."

WHAT IS ACTUALLY BEING SAID, AND WHETHER IT IS A QUESTION

  "limited to English"
     -> usually not a question. English-only is a resource limit,
        and "does it also work in German" is a chore unless there is
        a reason the mechanism should differ. Ask: does it? If the
        method depends on word order or morphology, now it is a
        question. Otherwise, skip.

  "models up to 7B"
     -> sometimes a question, and the good version is not "does it
        work at 70B". It is "does the effect grow or shrink with
        scale", because those two have opposite implications and
        three points on a curve distinguish them.

  "single retrieval corpus, do not vary the retriever"
     -> this is the question. The whole result may be a property of
        that corpus. Nobody can tell, including the authors, and one
        swap of the retriever answers it. Cheap, informative both
        ways, and nobody is racing you to it.

  "evaluation is automatic"
     -> a question only if you suspect the automatic metric is what
        the method is optimising. Check what the metric rewards. If
        it rewards length and the method produces longer outputs,
        that is now the most interesting sentence in the paper.

That ratio is normal and worth expecting. A limitation is a gap; a gap becomes a problem only when the answer would change what somebody does. The three discarded above fail that test, and reading them as opportunities is how people end up with a list of forty things and no direction.

Keep a problem list#

Write down every question you cannot answer, as you encounter it, without judging whether it is good. Most will be bad. The value is in the accumulation: after six months you have a list, and the patterns in it tell you what you actually care about, which is information you cannot get by introspection.

A WORKING ENTRY

  Q        does the curriculum-ordering effect survive a change of
           retriever, or is it a property of the one corpus used
  SOURCE   limitations para, the SFT-ordering paper, read 4 Mar
  MATTERS  if it is corpus-specific, the recommendation in that
           paper is wrong for everyone else, including us
  COST     one afternoon: same runs, second retriever
  ADVANTAGE  we already have both retrievers indexed
  KILL     if the delta is under 3 points either way, it is noise
           and I stop
  STATUS   untested

A list only becomes useful when entries can leave it. Give every entry a status and prune on a fixed schedule, otherwise the list grows monotonically and becomes a record of things you did not do. Four statuses are enough: untested, running, answered, and dropped-with-a-reason. The last one matters most, because a dropped entry with the reason attached stops you rediscovering the same bad idea in nine months and being pleased with it.

THE LIST, ONE QUARTER IN

  1  retriever swap on the ordering effect        untested   1 day
  2  does the length confound explain the agent
     failure result                               ANSWERED   yes, it
     did. Effect vanishes once length is controlled. Saved a month.
  3  do our eval scores move when the judge
     model version changes                        running    1 day
  4  attention entropy as a stuck-agent signal    DROPPED    someone
     published it in May, and better than I would have
  5  does chunk size interact with context
     length                                       untested   2 days
  6  can we predict tool-call failure from the
     schema alone                                 DROPPED    no
     decision waits on the answer. Interesting, not useful.

  QUARTERLY REVIEW: anything untested for two quarters is dropped.
  If it were important, it would have beaten something.

Track your surprises#

The practical difficulty is that a surprise decays. Within a day you will have constructed a reason it was not surprising, and the reason will feel like understanding. Writing it down at the moment it happens preserves the thing you actually predicted, which is the only part with information in it. An entry needs three lines and takes under a minute.

SURPRISE, 11 Apr

  I EXPECTED   raising retrieval top-k from 5 to 20 to improve answer
               accuracy, or at worst leave it flat
  IT DID       accuracy fell by 4 points. Latency tripled, as expected
  MY GUESS AT WHY
               the extra passages are mostly near-duplicates of the
               first five, and the model is being distracted rather
               than informed
  CHEAPEST CHECK
               dedupe the 20 passages and rerun. One hour.
  IF THE GUESS IS RIGHT
               top-k is the wrong knob and redundancy is the real
               variable, which would apply to every RAG system
               we have, not just this one

Most surprises are bugs, and that is fine — checking is cheap and finding your own bug is a good afternoon. But it is worth separating the two before spending a week, because the decision procedure differs. A bug is usually a surprise that is too large, too clean, or in the wrong direction for any mechanism you can name. A real finding is usually a surprise of a plausible size that survives when you change something irrelevant to it. Run the irrelevant change first: a different seed, a different subset, a different day. Surprises that do not survive that were never surprises.

The ones that do survive are worth more than anything on your problem list, because they are questions the field has not asked and you did not go looking for. You found them by accident, which means the probability that someone else is currently working on them is much lower than for anything you read about.

Read outside your area#

A method that is standard in one field and unknown in another is a recurring source of results. Reading adjacent literature is not a distraction; it is where the arbitrage is, and the history of machine learning is largely a history of borrowings.

• Markov chain Monte Carlo came into statistics from statistical physics, where sampling from a distribution nobody could integrate was already routine.
• Convolution came from signal processing, complete with the vocabulary of filters and kernels, before it was a layer.
• The Kalman filter was an aerospace and control technique long before it was a standard tool for tracking and state estimation elsewhere.
• Diffusion generative models were built on ideas from non-equilibrium thermodynamics, which is not a literature most machine-learning researchers were reading.
• Attention, having been developed for translation, moved into vision largely unchanged, which many people in vision did not expect to work.

The pattern in all five is the same and it tells you how to look. Another field had a mature answer to a problem your field was solving badly, and the barrier was vocabulary rather than difficulty. So the productive question when reading outside is not 'what have they found' but 'what do they do routinely that we do by hand, and what is their word for it'.

Concretely: once a quarter, read the methods sections — not the results — of three papers from a field one step away from yours. Statistics, control theory, experimental psychology, compilers, epidemiology, econometrics. You are looking for a standard technique with a name you did not know. Most quarters this yields nothing. The cost is an afternoon, and the payoff when it lands is a method that is decades mature arriving somewhere it has never been tried.

It is a different session from reading to learn#

The last practical point is that these two kinds of reading do not mix well, and trying to do both in one sitting produces neither. Reading to learn is receptive: you are trying to end up believing what the author believes, and interrupting yourself to look for holes makes you worse at it. Reading for direction is adversarial and much faster, because you are skipping most of the paper on purpose.

So separate them. Learn the area first, properly, over however many papers that takes. Then come back over the same papers with a different question and a notebook open, and do it in a block: six papers in ninety minutes, limitations and setup sections only, producing entries rather than understanding. The output of the second session is measured in list entries, and three or four from six papers is a good session. Zero is also a common one, and it means those six papers are further from your work than you thought — which is itself worth knowing before you spend a month there.