Research taste

Research taste is the judgement of which question is worth three months of your life. It is the hardest thing in research to teach and the thing senior hiring leans on hardest, because execution can be learned from almost anyone and this largely cannot. This section covers what the word means concretely, the signals that a problem is a good one, and how the judgement is actually developed — mostly by choosing badly a few times alongside someone who can say why.

What taste means operationally#

Executing an experiment is learnable. Choosing which experiment is worth running is not, in the same way, and it is what separates researchers whose work matters from researchers who are merely productive. But the word stays useless until you can say what someone with it actually does, so here is the operational version.

Given a candidate problem, taste is the ability to write three things down before starting: the outcomes you think are plausible and roughly how likely each is; what each outcome would license you or anyone else to do differently; and what it costs to find out. If you cannot write those three, you do not have a problem yet, you have an interest. That is not a criticism — most things start as an interest — but it tells you the next move is to sharpen it rather than to start work.

The reason this framing earns its place is that it makes the common failures visible as missing rows rather than as bad feelings. A problem whose outcomes are all one outcome is a demonstration, not an experiment. A problem where every outcome licenses the same next action is not worth running whatever it returns. And a problem whose cost you cannot state is one you have not thought about concretely enough to defend to anyone, including yourself in three months.

Signals#

• Tractable now. The field is full of important problems that cannot yet be attacked, and recognising which have just become attackable — because a tool, a dataset, an open model or a price changed — is most of the skill.
• The result matters either way. If a negative result would also be informative, the experiment is worth running, and you are insulated from the pressure to find something.
• Underexplored relative to importance. Crowded areas mean your marginal contribution is small, and you will be racing people with more compute.
• You have an unusual advantage: access, data, a deployed system, a background others lack. An advantage nobody else has is worth more than an idea nobody else has, because ideas are cheaper than access.
• It is a real question rather than an incremental variation on a benchmark number. The test: could the answer be interesting to someone who does not use your method?

Four candidates, judged#

Adjectives do not transfer. Judgements do. Here are four candidate problems of the ordinary kind — the sort that actually appear on a list, not famous open questions — run through the three rows above. The candidates are constructed, but the shape of each verdict is the shape the real ones take.

A. Does the standard 512-token chunk size for retrieval remain right
   when the reader model has a 200k context?

   OUTCOMES   larger chunks better (likely), no difference, worse
   LICENSES   a different default in every RAG system I touch, and a
              rule for when to change it
   COST       two days, one open model, our own query logs
   VERDICT    run it

B. A cheaper attention variant with better quality-per-FLOP.

   OUTCOMES   it does not work (overwhelmingly likely), it works
   LICENSES   if it works: nothing anyone can act on without a
              training run I cannot afford. If it does not: nothing
   COST       months, and compute I do not have
   VERDICT    drop. Crowded, no advantage, uninformative either way

C. Do agents fail more often on tickets that mention more than one
   system?

   OUTCOMES   yes, no, or it is really about ticket length
   LICENSES   a routing rule; but only if the third outcome is ruled
              out, which means length has to be controlled for
   COST       an afternoon on tickets already logged and labelled
   VERDICT    run it, after adding the length control

D. Apply the method I learned last month to this dataset.

   OUTCOMES   unclear, because no question has been asked
   LICENSES   nothing. There is no decision waiting on the answer
   COST       three weeks
   VERDICT    drop, and notice what happened: the method came first

A is the one to study, because it is unglamorous and it is the kind that pays. It is tractable specifically now — a long-context open model you can call cheaply did not exist a few years ago — the answer changes a default that is sitting in a great many systems, you have an advantage in the form of real query logs, and both outcomes are useful. Nobody will be impressed by the idea. That is close to irrelevant, and treating it as relevant is the single most common way good problems get passed over.

B fails on advantage rather than on interest. The question is genuinely important and that is precisely the problem: a large number of well-resourced people are on it, your marginal contribution is small, and the negative outcome — much the more likely one — teaches nobody anything, because 'my variant did not beat the baseline' is not a publishable or even a shareable fact. Important and crowded is the combination that wastes the most careers.

C is the useful case, because it is a bad problem that becomes a good one with one sentence of work. As stated it has a confound sitting in plain view: tickets mentioning several systems are longer, so any effect might be length. Adding the control converts an observation you could not act on into an experiment with three distinguishable outcomes, and it costs one extra column in the analysis. Most of your list will be in C's position rather than A's.

D is the failure worth naming explicitly: choosing a problem because a method is to hand rather than because the question matters. It produces work that is technically fine and that nobody needed, and it is much easier to fall into than it sounds, because it feels like momentum. The diagnostic is the order in which the sentence formed. If you can say what decision is waiting on the answer, you started from a question. If the method appears before the question does, you did not.

Sharpening rather than discarding#

Most candidates do not deserve a yes or a no on first reading; they deserve a rewrite. The rewrite is always in the same direction, from a topic towards a measurable claim with a named comparison. A worked sequence, each line a rewrite of the one above it:

v1  "retrieval quality in agents"
      - a topic, not a question. Nothing could come out false.

v2  "does better retrieval make agents better"
      - a question, but no quantity and no comparison. Better how,
        than what, measured on what?

v3  "does raising retrieval recall@5 from 0.6 to 0.8 reduce the
     agent's task failure rate"
      - measurable, but still unbounded: reduce by how much before
        I would call it an effect, and what is the noise floor?

v4  "on our 400 logged tickets, does swapping the retriever for one
     with recall@5 of 0.8 rather than 0.6 cut end-to-end failures by
     more than 3 points, which is the run-to-run spread we measured"
      - now it can come out false, the cost is an afternoon, and
        both answers change what we build next.

How it develops#

Taste is not acquired by reading about taste, which is a limitation this section shares with everything else written on the subject. It comes from making the judgement, recording it, and later finding out whether it was right — which requires that you wrote it down, because memory will revise it.

• Reading widely, including outside your area, so you notice what has just become possible rather than what is currently fashionable.
• Working alongside people who have it, which is the strongest argument for where you choose to work and is worth more than a title or a salary band early on.
• Being wrong in public repeatedly, and paying attention to why. A prediction you made and lost is worth a dozen you never wrote down.
• Following up on your own surprises. The thing that did not behave as you expected is a hole in your model of the field, and it is usually the direction.
• Keeping a scored list: every direction you considered, the verdict you gave it, and the date. Six months later, check. Two or three of those checks change how you judge the next one.

Why hiring assesses it#

Senior research interviews are largely an assessment of this. They ask what you would work on and why, what you think is overrated, what surprised you recently. There is no way to cram it, which is the point: the answers reveal how you actually think about the field.

What the interviewer is listening for is the three rows. A candidate who names an area is giving an interest. A candidate who names a question, says which way they expect it to go and why, and then says what they would do differently under each outcome, has demonstrated the thing in ninety seconds. It is also the reason a small, ordinary, well-judged problem answers this question better than an ambitious one: the ambitious one cannot be costed, and the costing is where the judgement shows.

This is reading done for a different purpose from the reading you have been doing until now. Instead of extracting what a paper found, you go through it for what it did not do: the ablation that is absent, the setting the method was never tested in, the limitation admitted in one sentence and dropped. The section covers that habit, the written record it produces, and what to do with a result that surprises you, which in practice is where a disproportionate number of real directions start.

Read for what is missing#

Most people read a paper to learn what was found. To find problems you read for what was not done: the ablation that is absent, the setting the method was never tested in, the limitation stated in one sentence and never revisited.

Limitations sections are the most under-read part of any paper and the most useful. Authors are telling you, honestly, where the work stops. That boundary is where the next question lives, and it is stated by the people best placed to know, usually because a reviewer made them state it.

Here is the operation performed rather than recommended. The limitations paragraph below is the ordinary kind — constructed, but you have read a hundred of these — followed by what comes out of it when you read it as a source of work rather than as a disclaimer.

THE PARAGRAPH, AS WRITTEN

  "Our experiments are limited to English and to models of up to 7B
   parameters. We use a single retrieval corpus and do not vary the
   retriever. Finally, our evaluation is automatic; we leave human
   evaluation to future work."

WHAT IS ACTUALLY BEING SAID, AND WHETHER IT IS A QUESTION

  "limited to English"
     -> usually not a question. English-only is a resource limit,
        and "does it also work in German" is a chore unless there is
        a reason the mechanism should differ. Ask: does it? If the
        method depends on word order or morphology, now it is a
        question. Otherwise, skip.

  "models up to 7B"
     -> sometimes a question, and the good version is not "does it
        work at 70B". It is "does the effect grow or shrink with
        scale", because those two have opposite implications and
        three points on a curve distinguish them.

  "single retrieval corpus, do not vary the retriever"
     -> this is the question. The whole result may be a property of
        that corpus. Nobody can tell, including the authors, and one
        swap of the retriever answers it. Cheap, informative both
        ways, and nobody is racing you to it.

  "evaluation is automatic"
     -> a question only if you suspect the automatic metric is what
        the method is optimising. Check what the metric rewards. If
        it rewards length and the method produces longer outputs,
        that is now the most interesting sentence in the paper.

That ratio is normal and worth expecting. A limitation is a gap; a gap becomes a problem only when the answer would change what somebody does. The three discarded above fail that test, and reading them as opportunities is how people end up with a list of forty things and no direction.

Keep a problem list#

Write down every question you cannot answer, as you encounter it, without judging whether it is good. Most will be bad. The value is in the accumulation: after six months you have a list, and the patterns in it tell you what you actually care about, which is information you cannot get by introspection.

A WORKING ENTRY

  Q        does the curriculum-ordering effect survive a change of
           retriever, or is it a property of the one corpus used
  SOURCE   limitations para, the SFT-ordering paper, read 4 Mar
  MATTERS  if it is corpus-specific, the recommendation in that
           paper is wrong for everyone else, including us
  COST     one afternoon: same runs, second retriever
  ADVANTAGE  we already have both retrievers indexed
  KILL     if the delta is under 3 points either way, it is noise
           and I stop
  STATUS   untested

A list only becomes useful when entries can leave it. Give every entry a status and prune on a fixed schedule, otherwise the list grows monotonically and becomes a record of things you did not do. Four statuses are enough: untested, running, answered, and dropped-with-a-reason. The last one matters most, because a dropped entry with the reason attached stops you rediscovering the same bad idea in nine months and being pleased with it.

THE LIST, ONE QUARTER IN

  1  retriever swap on the ordering effect        untested   1 day
  2  does the length confound explain the agent
     failure result                               ANSWERED   yes, it
     did. Effect vanishes once length is controlled. Saved a month.
  3  do our eval scores move when the judge
     model version changes                        running    1 day
  4  attention entropy as a stuck-agent signal    DROPPED    someone
     published it in May, and better than I would have
  5  does chunk size interact with context
     length                                       untested   2 days
  6  can we predict tool-call failure from the
     schema alone                                 DROPPED    no
     decision waits on the answer. Interesting, not useful.

  QUARTERLY REVIEW: anything untested for two quarters is dropped.
  If it were important, it would have beaten something.

Track your surprises#

The practical difficulty is that a surprise decays. Within a day you will have constructed a reason it was not surprising, and the reason will feel like understanding. Writing it down at the moment it happens preserves the thing you actually predicted, which is the only part with information in it. An entry needs three lines and takes under a minute.

SURPRISE, 11 Apr

  I EXPECTED   raising retrieval top-k from 5 to 20 to improve answer
               accuracy, or at worst leave it flat
  IT DID       accuracy fell by 4 points. Latency tripled, as expected
  MY GUESS AT WHY
               the extra passages are mostly near-duplicates of the
               first five, and the model is being distracted rather
               than informed
  CHEAPEST CHECK
               dedupe the 20 passages and rerun. One hour.
  IF THE GUESS IS RIGHT
               top-k is the wrong knob and redundancy is the real
               variable, which would apply to every RAG system
               we have, not just this one

Most surprises are bugs, and that is fine — checking is cheap and finding your own bug is a good afternoon. But it is worth separating the two before spending a week, because the decision procedure differs. A bug is usually a surprise that is too large, too clean, or in the wrong direction for any mechanism you can name. A real finding is usually a surprise of a plausible size that survives when you change something irrelevant to it. Run the irrelevant change first: a different seed, a different subset, a different day. Surprises that do not survive that were never surprises.

The ones that do survive are worth more than anything on your problem list, because they are questions the field has not asked and you did not go looking for. You found them by accident, which means the probability that someone else is currently working on them is much lower than for anything you read about.

Read outside your area#

A method that is standard in one field and unknown in another is a recurring source of results. Reading adjacent literature is not a distraction; it is where the arbitrage is, and the history of machine learning is largely a history of borrowings.

• Markov chain Monte Carlo came into statistics from statistical physics, where sampling from a distribution nobody could integrate was already routine.
• Convolution came from signal processing, complete with the vocabulary of filters and kernels, before it was a layer.
• The Kalman filter was an aerospace and control technique long before it was a standard tool for tracking and state estimation elsewhere.
• Diffusion generative models were built on ideas from non-equilibrium thermodynamics, which is not a literature most machine-learning researchers were reading.
• Attention, having been developed for translation, moved into vision largely unchanged, which many people in vision did not expect to work.

The pattern in all five is the same and it tells you how to look. Another field had a mature answer to a problem your field was solving badly, and the barrier was vocabulary rather than difficulty. So the productive question when reading outside is not 'what have they found' but 'what do they do routinely that we do by hand, and what is their word for it'.

Concretely: once a quarter, read the methods sections — not the results — of three papers from a field one step away from yours. Statistics, control theory, experimental psychology, compilers, epidemiology, econometrics. You are looking for a standard technique with a name you did not know. Most quarters this yields nothing. The cost is an afternoon, and the payoff when it lands is a method that is decades mature arriving somewhere it has never been tried.

It is a different session from reading to learn#

The last practical point is that these two kinds of reading do not mix well, and trying to do both in one sitting produces neither. Reading to learn is receptive: you are trying to end up believing what the author believes, and interrupting yourself to look for holes makes you worse at it. Reading for direction is adversarial and much faster, because you are skipping most of the paper on purpose.

So separate them. Learn the area first, properly, over however many papers that takes. Then come back over the same papers with a different question and a notebook open, and do it in a block: six papers in ninety minutes, limitations and setup sections only, producing entries rather than understanding. The output of the second session is measured in list entries, and three or four from six papers is a good session. Zero is also a common one, and it means those six papers are further from your work than you thought — which is itself worth knowing before you spend a month there.

Choosing a direction is only half the skill. The other half is dropping one quickly, and this section is about the mechanics of that: designing the cheapest experiment that could show the idea is wrong and running it before the careful implementation, writing down in advance what result would stop you, and separating the cases where a negative result is a verdict from the cases where it is a bug. The cost of not doing this is counted in months, and it is paid by people who were enthusiastic rather than careless.

Design the killing experiment first#

Before investing in a direction, ask what result would tell you it does not work, and how cheaply you could get that result. Running the falsifying experiment first is the highest-return habit in research, and it is the opposite of what enthusiasm suggests.

Idea:        method X should help on task Y

Wrong first step:   implement X properly, run the full benchmark
Right first step:   the crudest possible version on 100 examples

If the crude version shows nothing at all, the careful version
very rarely rescues it.

Written out on an ordinary idea, the difference between the two plans looks like this. The idea is that an agent which reviews its own tool call before issuing it will make fewer failing calls. It is plausible, it is cheap to believe, and it has a three-week version and a two-hour version.

IDEA  a self-review step before each tool call reduces failures

THE THREE-WEEK PLAN (what enthusiasm proposes)
  build the review step into the agent loop
  design a prompt for it, tune the prompt
  add a retry path for calls the reviewer rejects
  run the full 400-task suite, three seeds, with and without
  write it up
  -> three weeks before the first number

THE TWO-HOUR PLAN (what to do instead)
  take 200 tool calls already in the logs, of which 31 failed
  show each one to the model, cold, with the tool schema, and ask
    a single question: will this call succeed, yes or no
  count how often it catches the ones that failed

  THE POINT: if the model cannot tell a failing call from a working
  one when handed it in isolation, no amount of loop engineering
  will make it do so in flight. The whole idea rests on that one
  ability, so measure that one ability.

WHAT IT RETURNED
  caught 6 of the 31 real failures        (19% recall)
  flagged 44 of the 169 working calls     (26% false positive rate)
  -> a reviewer that rejects a quarter of good calls to catch a
     fifth of bad ones. The retry path would cost more than the
     failures do.

DECISION  dropped, after two hours, with a number rather than a
          feeling. The write-up is four lines long and it is the
          most useful thing on the problem list this month.

Notice what the two-hour version skipped: the loop, the retry path, the prompt tuning, the full suite. All of those are engineering that would have been real work, and none of them could have rescued a reviewer that cannot distinguish a good call from a bad one when the question is handed to it directly. That is the general recipe. Find the one assumption the whole direction rests on, strip away everything that is not that assumption, and measure it on data you already have.

The objection that always arises is that the crude version is unfair to the idea, and sometimes it is. The answer is not to skip it but to say in advance what the crude version has to show for the careful one to be worth building. If you cannot name that number beforehand, the crude experiment was never going to change your mind and you should admit that before running it rather than afterwards.

Write down what would change your mind#

Decide in advance what result would make you stop. Written before you see data, it is a genuine commitment. Decided afterwards, you will find a reason the number does not count, and you will be sincere about it.

It takes ten minutes and it fits on a screen. The value is entirely in the timestamp: this is a promise made by someone who is not yet invested, to the person who will be invested in six weeks and will not be able to see straight.

PRE-REGISTRATION, 2 May, before any runs

CLAIM        reranking the retrieved passages reduces answer errors
QUANTITY     exact-match accuracy on our 400 held-out questions
COMPARISON   same pipeline, reranker removed. Same retriever, same
             top-k into the reader, same prompt.
NOISE FLOOR  measured first, before the treatment: three seeds of
             the baseline gave 71.2 / 72.0 / 72.6, so a spread of
             1.4 points. Anything under that is nothing.

I STOP IF    the gain is under 1.4 points averaged over three seeds
I CONTINUE IF the gain is over 3 points, and then the next thing I
             do is the latency measurement, not more accuracy work
IN BETWEEN   1.4 to 3 points: run three more seeds, once. If it is
             still in between, stop. An effect I need six seeds to
             see is not an effect anyone will benefit from.

WHAT I EXPECT  about 2 points. I am writing this down so that I
             cannot later claim I predicted whatever happened.

NOT ALLOWED  changing the eval set, changing the metric, dropping
             the seeds I dislike, or deciding the held-out set was
             unrepresentative after seeing the numbers.

Measuring the noise floor before the treatment is the non-obvious discipline in that document, and skipping it is how most pre-registrations become decorative. If you set the stopping threshold after you know the spread, you will set it just below whatever you got. Measure the baseline variance first, write the threshold from it, and you have bound yourself with a number that came from the world rather than from your hopes.

Sunk cost#

This is easy to agree with and hard to apply, because sunk cost never presents itself as sunk cost. It arrives as a technical argument. Here is the same situation reasoned both ways, five months into a direction that has not produced a positive result.

THE ARGUMENT THAT FEELS LIKE ANALYSIS

  "We are five months in and the infrastructure is finally working.
   The last three failures were all setup problems. It would be
   mad to walk away now, right when we can finally run things
   properly - the real experiment has not even started."

  Every clause is true. None of it is evidence about whether the
  hypothesis is correct. Read it again with that in mind: it argues
  entirely from cost and readiness, never from a result.

THE FORWARD CALCULATION

  What do I now believe, that I did not believe in December?
    - the effect, if it exists, is smaller than 2 points, because
      we would have seen a larger one through the setup noise
    - the setting where it should be strongest is the one where we
      measured least, and that was not an accident: it is hardest

  What would the next month buy?
    - one clean run of the experiment we have wanted since December
    - probability I would put on a positive result: maybe 15%

  What else could the next month buy?
    - three of the afternoon-sized questions on the list, two of
      which change a default in a system we ship

  DECISION  one clean run, timeboxed to three weeks, because the
  experiment is now cheap and an unanswered question is worth
  closing. Then stop regardless of the outcome's ambiguity. The
  five months are not in this calculation anywhere.

The distinction that survives is this one: what you have already spent is irrelevant, but what you have already built is an asset, and an asset changes the forward cost. Five months of belief is worth nothing. Five months of working infrastructure genuinely makes the next experiment cheaper, and cheapness is a legitimate reason to run it. The failure is using the first to argue for the second.

When to persist#

Not every negative result is a verdict. Persist when the failure is clearly mechanical rather than conceptual, when you can name specifically what went wrong, or when the idea keeps being independently suggested by people you respect. Abandon when you are persisting because stopping would feel like failure.

The distinction is sharper than it sounds, and the test is whether you can name the fault before you fix it. Two cases that look identical from the outside — a flat result, an enthusiastic researcher — and separate cleanly under that test.

CASE 1  MECHANICAL. Persist.

  Result      fine-tuning made no difference at all. Exactly none:
              the tuned model's outputs are identical to the base
              model's on 400 of 400 examples.
  Diagnostic  "identical on all 400" is not what a real null result
              looks like. A method that does not help still moves
              outputs around.
  Named fault I can state it before fixing it: the adapter weights
              are probably not being loaded at inference.
  Cost        twenty minutes to check
  Verdict     this is a bug, not a finding. Fix it, then re-judge.

CASE 2  CONCEPTUAL. Stop.

  Result      the method helps by 0.4 points, seed spread 1.1, on
              the fourth variant of the setup we have tried
  Diagnostic  each variant was chosen after the previous one failed,
              and each was justified afterwards. Nothing was
              pre-registered after the first one.
  Named fault none. The explanation for each failure was different
              and arrived after the number did.
  Verdict     stop. Four post-hoc explanations is not four pieces
              of evidence, it is one piece of evidence about me.

THE TEST  can you name the fault BEFORE you fix it, and does the
fix have an independent reason to be right? One yes is persistence.
Two nos is a search for a setting where the noise is kind.

There is a third case where persistence is right and neither test applies: the idea keeps being independently proposed by people whose judgement you rate, and nobody has actually tested it. That is weak evidence about the hypothesis, but it is strong evidence that the answer is wanted, which means a clean negative result has a reader. Under those conditions the experiment is worth finishing even when you expect it to fail, and you should say that out loud in advance so the failure is not a disappointment when it arrives.

A research pitch is one page arguing that a question is worth someone's time. It is the artefact that gets a direction funded, staffed or killed, and writing one is the fastest way to discover that an idea you have been carrying for a month does not survive contact with a page. This section gives the questions a pitch has to answer, then writes one out in full, because the advice to be clear about your contribution is useless and the paragraph that does it is not.

The questions a pitch must answer#

Five, in this order. A reader who cannot find all five in under two minutes will decline, and usually without telling you which one was missing.

1. what is the question          stated so it could be answered yes or no
2. why does it matter            what changes if the answer is yes
3. why is it open                why the obvious approaches have not settled it
4. what is the first experiment  small, specific, and doable in weeks
5. what would change your mind   the result that would make you stop

One written out#

Here is the whole thing, at the length it should be. The subject is deliberately ordinary — it is not a famous open problem, and the pitch is stronger for that.

Does retrieval quality degrade when a model is fine-tuned on its own
retrieved outputs?
-------------------------------------------------------------------

THE QUESTION.  Systems that fine-tune on their own logged outputs are now
common, and those outputs were produced with retrieved context. If the
model learns to rely on information that was in the context rather than
in its weights, its behaviour without that context should get worse. Does
it, and by how much?

WHY IT MATTERS.  If it does, the standard loop of logging production
traffic and fine-tuning on it quietly degrades the model for every path
that does not retrieve -- which for most products is the majority of
calls. Nobody would see it, because the eval set is drawn from the same
retrieved-context distribution.

WHY IT IS OPEN.  The obvious check is to evaluate with and without
context, and papers on self-training do not report the second number. The
closest work measures hallucination after self-training but always with
retrieval present. So the effect could be large, zero, or reversed, and I
cannot find the experiment.

FIRST EXPERIMENT.  Two weeks, one GPU. Take a 7B instruct model, generate
answers to 20,000 questions with retrieval, fine-tune on them, and
evaluate the tuned and untuned models on held-out questions BOTH with and
without context. The quantity of interest is the change in the no-context
score. Baseline: the same fine-tune on the same answers with the context
stripped from the prompt.

WHAT WOULD CHANGE MY MIND.  If the no-context score moves by less than
the seed-to-seed spread (about a point on this setup, measured over three
seeds), there is no effect worth chasing and I stop. If it drops by more
than three points I will run the ablation on context length, since the
mechanism predicts a larger effect with more retrieved text.

RISKS.  The effect may be specific to one model family; I will confirm on
a second before writing anything up.

Read the third paragraph again, because it is the one people write badly. It does not claim nobody has thought about this; it says precisely what the nearest work measured and what it did not, which is a checkable statement rather than a boast. A reader who knows the literature can correct you in one line, and that correction is worth more than the month you would otherwise spend.

The smallest experiment first#

The first experiment in a pitch should be embarrassingly small — small enough that you would run it this fortnight rather than after a planning cycle. Its job is not to answer the question but to establish that the effect exists and is measurable at all. Most pitches die here for a good reason: the author cannot name an experiment under a month, which usually means the question is not yet sharp enough to test.

• Prefer the smallest model and dataset that could show the effect. If it only appears at scale, say so and explain how you know.
• Name the quantity you will measure, not the thing you will build.
• State the baseline in the same sentence as the treatment, or the result will be uninterpretable.
• Give the number the result has to beat, which means knowing your noise floor before you start.
• Cost it in days and GPUs. A pitch without a number here reads as wishful.

What result would change your mind#

Write this before running anything, and write it as a number rather than a direction. 'If it gets worse I will reconsider' commits you to nothing; 'if the no-context score moves by less than a point, which is the seed spread here, I stop' commits you to something you cannot argue your way out of at the time. That is the entire function of the sentence: it is a promise made by someone who is not yet invested to someone who will be.

Getting useful criticism#

Send the page to two people, and ask each a specific question rather than for thoughts. Ask someone who knows the area whether the third paragraph is true — whether the work you claim does not exist really does not exist. Ask someone who does not know the area whether they can restate your question back to you, because if they cannot, the first paragraph has failed and nothing after it will be read.

The useful response to either is not defence. If the literature reader names a paper that already did it, that is the pitch working: it cost you an afternoon instead of a quarter. And a direction that survives both readings is one you can start on knowing that the fastest ways of being wrong have already been tried.

Taste in research is largely the ability to tell which question is worth asking now, and it can be studied directly by looking at how well-known work was chosen rather than at what it found. This section is a reading method: for a paper you admire, reconstruct what the field believed the day before it appeared, what had recently changed, and why nobody else had taken the step. Doing the same for directions that attracted effort and faded is the half people skip, and it is the half that corrects for hindsight.

Reverse-engineer the choice#

Reading famous papers for their results teaches you results. Reading them to reconstruct why the authors chose that problem at that moment teaches you taste. For any paper you admire, try to answer four questions using only what was published before it.

1. 1.What did the field believe just before this appeared?
2. 2.What had recently changed: data, compute, a tool, a related result?
3. 3.Why was this the next step, and why had nobody taken it?
4. 4.What would a failed version of this project have looked like?

The discipline is to answer them with what was published before, not with what you know now. The answers below are reconstructions of publicly documented situations; they are arguments about timing rather than claims about what anyone was thinking.

Case one: removing recurrence, 2017#

What the field believed. The best translation systems were recurrent encoder-decoders with attention bolted on. Attention had been introduced a few years earlier to let the decoder look back at the whole source sentence instead of a single compressed vector, and it worked, but it was understood as an addition to recurrence. Recurrence was what carried the sequence: the model saw position three after position two because the computation was literally ordered that way. Convolutional alternatives for sequence-to-sequence had been published and were taken seriously, so the monopoly of recurrence was already being questioned; nobody had proposed removing both.

What had recently changed. Two things, and neither was an idea. Attention had accumulated enough evidence that it was doing a large share of the work rather than assisting. And the hardware had moved: accelerators were getting much faster at large dense matrix multiplications, while recurrence is sequential along the sequence and so cannot use that width. The cost of recurrence was therefore rising every year, not because recurrence got worse but because the alternative got cheaper. A constraint that shifts underneath a field without any paper announcing it is the most common form of the answer to question two.

Why it was a good bet. The question was informative in both directions, which is the property to look for. If attention alone matched the recurrent systems, training could be parallelised across positions and everything downstream would get cheaper. If it did not, the field would have learned something specific about what recurrence contributes that attention does not — which was, at the time, genuinely unknown rather than rhetorical.

Why others had not taken it. Partly the belief that order information had to come from ordered computation. Removing recurrence means the model can no longer tell position three from position seven, and the fix — adding a representation of position to the input — looks like a hack rather than a solution until it works. The unobvious step was not attention. It was being willing to patch the hole that removing recurrence opened, instead of treating the hole as proof that the idea was wrong.

The failed version. Attention-only underperforms, positional information proves insufficient without recurrence, and the paper is a short negative result showing where attention runs out. That paper would still have been worth writing, and the fact that you can say so is the test of whether the bet was good independently of how it landed.

Case two: learned features beating designed ones, 2012#

What the field believed. Computer vision was a discipline of designed features. You built a descriptor by hand, aggregated it, and put a classifier on top; the intellectual content of the field was in the descriptor. Convolutional networks had existed since the late 1980s and had been genuinely deployed on constrained problems such as reading digits, so nobody thought they did not work. The belief was that they did not scale to natural images, and that belief had been supported by a decade of results.

What had recently changed. Three things at once, which is the usual pattern for a step this large. A labelled dataset of over a million natural images across a thousand categories existed, where previously the standard benchmarks had tens of thousands of images. General-purpose programming of graphics hardware had matured enough that a large convolutional network could be trained in days rather than months. And a handful of training details — a non-saturating activation, a regularisation scheme aggressive enough for a model that size — had made networks of that depth trainable at all.

Why it was a good bet. The hypothesis was not 'neural networks are good'. It was much narrower and much better posed: that the previous failures were a data and compute limitation rather than a limitation of the approach. That is a claim about which constraint is binding, and the new dataset plus the new hardware were precisely an opportunity to test it. Either outcome answers a question the field had been arguing about for years.

Why others had not taken it. The dataset was new, the engineering was hard and unfashionable, and the prior was against it: proposing to spend months on a method with a long record of disappointing at scale is a reputationally expensive thing to do. There is also a skills point that recurs in these reconstructions. The step required someone who could write fast GPU code and who believed the scaling argument, and the intersection of those two populations was small. An advantage does not have to be an insight; often it is just a combination of capabilities that happens to be rare.

The failed version. The large network performs about as well as the best feature pipelines, and the entry becomes another data point that neural networks need far more data and compute for no real gain. The direction quietly costs its authors two years. Written down beforehand, that outcome was clearly on the table, and the bet was good anyway because the margin of the win and the margin of the loss were both informative.

Case three: a direction that attracted effort and faded#

Survivorship bias is strong in research history, and studying only successes teaches you to recognise winners after they have won. The corrective is to run the same four questions on a direction that drew serious work from serious people and did not become part of standard practice. Capsule networks, from 2017, are a good subject precisely because the argument behind them was a good argument.

What was believed, and what the direction objected to. The criticism was long-standing and substantive: pooling in convolutional networks discards precise spatial relationships between parts, so a network can learn to recognise a face from the presence of two eyes and a mouth without representing how they are arranged. The proposal was to represent entities as groups of units carrying pose information, with a routing procedure deciding which lower-level parts belong to which higher-level whole. It came from people with a long record, it addressed a real defect, and it was taken up widely enough to generate a substantial follow-on literature within a couple of years.

What signals were visible at the time. Three, and all three were visible in the original work rather than in hindsight. First, the headline results were on small image datasets, and a method whose motivation is about complex natural scenes being demonstrated on digits is a gap between argument and evidence. Second, the routing procedure is iterative and its cost does not fall as hardware widens, which sets it against the direction the compute was moving — the same constraint that made case one work, running the other way. Third, and most transferable: the direction was motivated by a defect in the incumbent rather than by a measured bottleneck on a task anyone cared about. Nobody had shown that the spatial-relationship failure was what was limiting performance in practice.

That third signal is the one worth carrying to your own ideas, and it is the most uncomfortable, because 'the current approach has a flaw I can name' is exactly what a good research idea feels like from the inside. It is not sufficient. A flaw becomes a direction when you can show it is the binding constraint on something measurable, and the cheap version of that demonstration usually exists. Here it would have been an experiment showing a task where performance is limited specifically by discarded spatial relationships, run with the incumbent architecture, before proposing a replacement for it.

A second, more ordinary example of the same shape: the long run of neural architectures proposed for tabular prediction. Tabular data is enormously common, the argument that representation learning should help was reasonable, and a great deal of work was done. Gradient-boosted trees remained the thing practitioners reach for. The signal available throughout was that the reported gains were small and setting-dependent while the incumbent was cheap, tuned in minutes, and understood — and a direction that must win narrowly against a cheap, well-understood baseline is a direction with very little room.

What the three have in common#

Lay them side by side and the same variable decides all three. Not the quality of the idea, which was high in every case including the one that faded, but whether an external constraint had recently moved in the direction the idea needed.

                 WHAT MOVED                 DIRECTION OF THE MOVE

removing         hardware widened; the cost   with the idea. Every
recurrence       of sequential computation    year made it more
                 rose relative to parallel    right, not less

learned          a million-image labelled     with the idea. The
features         dataset appeared; GPU        binding constraint had
                 training became practical    just been removed

capsules         compute kept widening, and   against the idea. An
                 the field moved towards      iterative routing step
                 scaling simple, parallel     gets relatively more
                 architectures                expensive over time

tabular          nothing moved. The           neutral, which against
networks         incumbent stayed cheap,      a cheap incumbent is
                 fast and well understood     the same as against

Then turn the four questions on yourself, which is the point of the exercise. What does the field currently believe about your area, stated as a sentence somebody would disagree with? What changed in the last eighteen months — a price, an open model, a dataset, a tool — that was not true before? Why has nobody done this, and is the honest answer that it is hard, or that it is unfashionable, or that they have and you have not found it? And what would the failed version look like, written out, so that you find out now whether you would be willing to publish it.