Learning taste from how good work was chosen

Taste in research is largely the ability to tell which question is worth asking now, and it can be studied directly by looking at how well-known work was chosen rather than at what it found. This section is a reading method: for a paper you admire, reconstruct what the field believed the day before it appeared, what had recently changed, and why nobody else had taken the step. Doing the same for directions that attracted effort and faded is the half people skip, and it is the half that corrects for hindsight.

Reverse-engineer the choice#

Reading famous papers for their results teaches you results. Reading them to reconstruct why the authors chose that problem at that moment teaches you taste. For any paper you admire, try to answer four questions using only what was published before it.

1. 1.What did the field believe just before this appeared?
2. 2.What had recently changed: data, compute, a tool, a related result?
3. 3.Why was this the next step, and why had nobody taken it?
4. 4.What would a failed version of this project have looked like?

The discipline is to answer them with what was published before, not with what you know now. The answers below are reconstructions of publicly documented situations; they are arguments about timing rather than claims about what anyone was thinking.

Case one: removing recurrence, 2017#

What the field believed. The best translation systems were recurrent encoder-decoders with attention bolted on. Attention had been introduced a few years earlier to let the decoder look back at the whole source sentence instead of a single compressed vector, and it worked, but it was understood as an addition to recurrence. Recurrence was what carried the sequence: the model saw position three after position two because the computation was literally ordered that way. Convolutional alternatives for sequence-to-sequence had been published and were taken seriously, so the monopoly of recurrence was already being questioned; nobody had proposed removing both.

What had recently changed. Two things, and neither was an idea. Attention had accumulated enough evidence that it was doing a large share of the work rather than assisting. And the hardware had moved: accelerators were getting much faster at large dense matrix multiplications, while recurrence is sequential along the sequence and so cannot use that width. The cost of recurrence was therefore rising every year, not because recurrence got worse but because the alternative got cheaper. A constraint that shifts underneath a field without any paper announcing it is the most common form of the answer to question two.

Why it was a good bet. The question was informative in both directions, which is the property to look for. If attention alone matched the recurrent systems, training could be parallelised across positions and everything downstream would get cheaper. If it did not, the field would have learned something specific about what recurrence contributes that attention does not — which was, at the time, genuinely unknown rather than rhetorical.

Why others had not taken it. Partly the belief that order information had to come from ordered computation. Removing recurrence means the model can no longer tell position three from position seven, and the fix — adding a representation of position to the input — looks like a hack rather than a solution until it works. The unobvious step was not attention. It was being willing to patch the hole that removing recurrence opened, instead of treating the hole as proof that the idea was wrong.

The failed version. Attention-only underperforms, positional information proves insufficient without recurrence, and the paper is a short negative result showing where attention runs out. That paper would still have been worth writing, and the fact that you can say so is the test of whether the bet was good independently of how it landed.

Case two: learned features beating designed ones, 2012#

What the field believed. Computer vision was a discipline of designed features. You built a descriptor by hand, aggregated it, and put a classifier on top; the intellectual content of the field was in the descriptor. Convolutional networks had existed since the late 1980s and had been genuinely deployed on constrained problems such as reading digits, so nobody thought they did not work. The belief was that they did not scale to natural images, and that belief had been supported by a decade of results.

What had recently changed. Three things at once, which is the usual pattern for a step this large. A labelled dataset of over a million natural images across a thousand categories existed, where previously the standard benchmarks had tens of thousands of images. General-purpose programming of graphics hardware had matured enough that a large convolutional network could be trained in days rather than months. And a handful of training details — a non-saturating activation, a regularisation scheme aggressive enough for a model that size — had made networks of that depth trainable at all.

Why it was a good bet. The hypothesis was not 'neural networks are good'. It was much narrower and much better posed: that the previous failures were a data and compute limitation rather than a limitation of the approach. That is a claim about which constraint is binding, and the new dataset plus the new hardware were precisely an opportunity to test it. Either outcome answers a question the field had been arguing about for years.

Why others had not taken it. The dataset was new, the engineering was hard and unfashionable, and the prior was against it: proposing to spend months on a method with a long record of disappointing at scale is a reputationally expensive thing to do. There is also a skills point that recurs in these reconstructions. The step required someone who could write fast GPU code and who believed the scaling argument, and the intersection of those two populations was small. An advantage does not have to be an insight; often it is just a combination of capabilities that happens to be rare.

The failed version. The large network performs about as well as the best feature pipelines, and the entry becomes another data point that neural networks need far more data and compute for no real gain. The direction quietly costs its authors two years. Written down beforehand, that outcome was clearly on the table, and the bet was good anyway because the margin of the win and the margin of the loss were both informative.

Case three: a direction that attracted effort and faded#

Survivorship bias is strong in research history, and studying only successes teaches you to recognise winners after they have won. The corrective is to run the same four questions on a direction that drew serious work from serious people and did not become part of standard practice. Capsule networks, from 2017, are a good subject precisely because the argument behind them was a good argument.

What was believed, and what the direction objected to. The criticism was long-standing and substantive: pooling in convolutional networks discards precise spatial relationships between parts, so a network can learn to recognise a face from the presence of two eyes and a mouth without representing how they are arranged. The proposal was to represent entities as groups of units carrying pose information, with a routing procedure deciding which lower-level parts belong to which higher-level whole. It came from people with a long record, it addressed a real defect, and it was taken up widely enough to generate a substantial follow-on literature within a couple of years.

What signals were visible at the time. Three, and all three were visible in the original work rather than in hindsight. First, the headline results were on small image datasets, and a method whose motivation is about complex natural scenes being demonstrated on digits is a gap between argument and evidence. Second, the routing procedure is iterative and its cost does not fall as hardware widens, which sets it against the direction the compute was moving — the same constraint that made case one work, running the other way. Third, and most transferable: the direction was motivated by a defect in the incumbent rather than by a measured bottleneck on a task anyone cared about. Nobody had shown that the spatial-relationship failure was what was limiting performance in practice.

That third signal is the one worth carrying to your own ideas, and it is the most uncomfortable, because 'the current approach has a flaw I can name' is exactly what a good research idea feels like from the inside. It is not sufficient. A flaw becomes a direction when you can show it is the binding constraint on something measurable, and the cheap version of that demonstration usually exists. Here it would have been an experiment showing a task where performance is limited specifically by discarded spatial relationships, run with the incumbent architecture, before proposing a replacement for it.

A second, more ordinary example of the same shape: the long run of neural architectures proposed for tabular prediction. Tabular data is enormously common, the argument that representation learning should help was reasonable, and a great deal of work was done. Gradient-boosted trees remained the thing practitioners reach for. The signal available throughout was that the reported gains were small and setting-dependent while the incumbent was cheap, tuned in minutes, and understood — and a direction that must win narrowly against a cheap, well-understood baseline is a direction with very little room.

What the three have in common#

Lay them side by side and the same variable decides all three. Not the quality of the idea, which was high in every case including the one that faded, but whether an external constraint had recently moved in the direction the idea needed.

                 WHAT MOVED                 DIRECTION OF THE MOVE

removing         hardware widened; the cost   with the idea. Every
recurrence       of sequential computation    year made it more
                 rose relative to parallel    right, not less

learned          a million-image labelled     with the idea. The
features         dataset appeared; GPU        binding constraint had
                 training became practical    just been removed

capsules         compute kept widening, and   against the idea. An
                 the field moved towards      iterative routing step
                 scaling simple, parallel     gets relatively more
                 architectures                expensive over time

tabular          nothing moved. The           neutral, which against
networks         incumbent stayed cheap,      a cheap incumbent is
                 fast and well understood     the same as against

Then turn the four questions on yourself, which is the point of the exercise. What does the field currently believe about your area, stated as a sentence somebody would disagree with? What changed in the last eighteen months — a price, an open model, a dataset, a tool — that was not true before? Why has nobody done this, and is the honest answer that it is hard, or that it is unfashionable, or that they have and you have not found it? And what would the failed version look like, written out, so that you find out now whether you would be willing to publish it.