Choosing a problem worth solving

Research taste is the judgement of which question is worth three months of your life. It is the hardest thing in research to teach and the thing senior hiring leans on hardest, because execution can be learned from almost anyone and this largely cannot. This section covers what the word means concretely, the signals that a problem is a good one, and how the judgement is actually developed — mostly by choosing badly a few times alongside someone who can say why.

What taste means operationally#

Executing an experiment is learnable. Choosing which experiment is worth running is not, in the same way, and it is what separates researchers whose work matters from researchers who are merely productive. But the word stays useless until you can say what someone with it actually does, so here is the operational version.

Given a candidate problem, taste is the ability to write three things down before starting: the outcomes you think are plausible and roughly how likely each is; what each outcome would license you or anyone else to do differently; and what it costs to find out. If you cannot write those three, you do not have a problem yet, you have an interest. That is not a criticism — most things start as an interest — but it tells you the next move is to sharpen it rather than to start work.

The reason this framing earns its place is that it makes the common failures visible as missing rows rather than as bad feelings. A problem whose outcomes are all one outcome is a demonstration, not an experiment. A problem where every outcome licenses the same next action is not worth running whatever it returns. And a problem whose cost you cannot state is one you have not thought about concretely enough to defend to anyone, including yourself in three months.

Signals#

• Tractable now. The field is full of important problems that cannot yet be attacked, and recognising which have just become attackable — because a tool, a dataset, an open model or a price changed — is most of the skill.
• The result matters either way. If a negative result would also be informative, the experiment is worth running, and you are insulated from the pressure to find something.
• Underexplored relative to importance. Crowded areas mean your marginal contribution is small, and you will be racing people with more compute.
• You have an unusual advantage: access, data, a deployed system, a background others lack. An advantage nobody else has is worth more than an idea nobody else has, because ideas are cheaper than access.
• It is a real question rather than an incremental variation on a benchmark number. The test: could the answer be interesting to someone who does not use your method?

Four candidates, judged#

Adjectives do not transfer. Judgements do. Here are four candidate problems of the ordinary kind — the sort that actually appear on a list, not famous open questions — run through the three rows above. The candidates are constructed, but the shape of each verdict is the shape the real ones take.

A. Does the standard 512-token chunk size for retrieval remain right
   when the reader model has a 200k context?

   OUTCOMES   larger chunks better (likely), no difference, worse
   LICENSES   a different default in every RAG system I touch, and a
              rule for when to change it
   COST       two days, one open model, our own query logs
   VERDICT    run it

B. A cheaper attention variant with better quality-per-FLOP.

   OUTCOMES   it does not work (overwhelmingly likely), it works
   LICENSES   if it works: nothing anyone can act on without a
              training run I cannot afford. If it does not: nothing
   COST       months, and compute I do not have
   VERDICT    drop. Crowded, no advantage, uninformative either way

C. Do agents fail more often on tickets that mention more than one
   system?

   OUTCOMES   yes, no, or it is really about ticket length
   LICENSES   a routing rule; but only if the third outcome is ruled
              out, which means length has to be controlled for
   COST       an afternoon on tickets already logged and labelled
   VERDICT    run it, after adding the length control

D. Apply the method I learned last month to this dataset.

   OUTCOMES   unclear, because no question has been asked
   LICENSES   nothing. There is no decision waiting on the answer
   COST       three weeks
   VERDICT    drop, and notice what happened: the method came first

A is the one to study, because it is unglamorous and it is the kind that pays. It is tractable specifically now — a long-context open model you can call cheaply did not exist a few years ago — the answer changes a default that is sitting in a great many systems, you have an advantage in the form of real query logs, and both outcomes are useful. Nobody will be impressed by the idea. That is close to irrelevant, and treating it as relevant is the single most common way good problems get passed over.

B fails on advantage rather than on interest. The question is genuinely important and that is precisely the problem: a large number of well-resourced people are on it, your marginal contribution is small, and the negative outcome — much the more likely one — teaches nobody anything, because 'my variant did not beat the baseline' is not a publishable or even a shareable fact. Important and crowded is the combination that wastes the most careers.

C is the useful case, because it is a bad problem that becomes a good one with one sentence of work. As stated it has a confound sitting in plain view: tickets mentioning several systems are longer, so any effect might be length. Adding the control converts an observation you could not act on into an experiment with three distinguishable outcomes, and it costs one extra column in the analysis. Most of your list will be in C's position rather than A's.

D is the failure worth naming explicitly: choosing a problem because a method is to hand rather than because the question matters. It produces work that is technically fine and that nobody needed, and it is much easier to fall into than it sounds, because it feels like momentum. The diagnostic is the order in which the sentence formed. If you can say what decision is waiting on the answer, you started from a question. If the method appears before the question does, you did not.

Sharpening rather than discarding#

Most candidates do not deserve a yes or a no on first reading; they deserve a rewrite. The rewrite is always in the same direction, from a topic towards a measurable claim with a named comparison. A worked sequence, each line a rewrite of the one above it:

v1  "retrieval quality in agents"
      - a topic, not a question. Nothing could come out false.

v2  "does better retrieval make agents better"
      - a question, but no quantity and no comparison. Better how,
        than what, measured on what?

v3  "does raising retrieval recall@5 from 0.6 to 0.8 reduce the
     agent's task failure rate"
      - measurable, but still unbounded: reduce by how much before
        I would call it an effect, and what is the noise floor?

v4  "on our 400 logged tickets, does swapping the retriever for one
     with recall@5 of 0.8 rather than 0.6 cut end-to-end failures by
     more than 3 points, which is the run-to-run spread we measured"
      - now it can come out false, the cost is an afternoon, and
        both answers change what we build next.

How it develops#

Taste is not acquired by reading about taste, which is a limitation this section shares with everything else written on the subject. It comes from making the judgement, recording it, and later finding out whether it was right — which requires that you wrote it down, because memory will revise it.

• Reading widely, including outside your area, so you notice what has just become possible rather than what is currently fashionable.
• Working alongside people who have it, which is the strongest argument for where you choose to work and is worth more than a title or a salary band early on.
• Being wrong in public repeatedly, and paying attention to why. A prediction you made and lost is worth a dozen you never wrote down.
• Following up on your own surprises. The thing that did not behave as you expected is a hole in your model of the field, and it is usually the direction.
• Keeping a scored list: every direction you considered, the verdict you gave it, and the date. Six months later, check. Two or three of those checks change how you judge the next one.

Why hiring assesses it#

Senior research interviews are largely an assessment of this. They ask what you would work on and why, what you think is overrated, what surprised you recently. There is no way to cram it, which is the point: the answers reveal how you actually think about the field.

What the interviewer is listening for is the three rows. A candidate who names an area is giving an interest. A candidate who names a question, says which way they expect it to go and why, and then says what they would do differently under each outcome, has demonstrated the thing in ninety seconds. It is also the reason a small, ordinary, well-judged problem answers this question better than an ambitious one: the ambitious one cannot be costed, and the costing is where the judgement shows.