Killing your own ideas early
By the end: You can abandon a direction before it costs you six months.
- The cheapest experiment that could falsify it
- Pre-registering what would change your mind
- Sunk cost in research
- When persistence is right
New to research taste? The name sounds mystical and the activity is not. Research taste is judgement about which question to work on next, and it comes down to questions asked before any work starts: is this actually still open, would anyone do anything differently if I answered it, can it be answered with the compute and data I can get, and is there a small version I can run in a week that would tell me I am wrong. Why it matters
Choosing a direction is only half the skill. The other half is dropping one quickly, and this section is about the mechanics of that: designing the cheapest experiment that could show the idea is wrong and running it before the careful implementation, writing down in advance what result would stop you, and separating the cases where a negative result is a verdict from the cases where it is a bug. The cost of not doing this is counted in months, and it is paid by people who were enthusiastic rather than careless.
Design the killing experiment first#
Before investing in a direction, ask what result would tell you it does not work, and how cheaply you could get that result. Running the falsifying experiment first is the highest-return habit in research, and it is the opposite of what enthusiasm suggests.
Idea: method X should help on task Y
Wrong first step: implement X properly, run the full benchmark
Right first step: the crudest possible version on 100 examples
If the crude version shows nothing at all, the careful version
very rarely rescues it.Written out on an ordinary idea, the difference between the two plans looks like this. The idea is that an agent which reviews its own tool call before issuing it will make fewer failing calls. It is plausible, it is cheap to believe, and it has a three-week version and a two-hour version.
IDEA a self-review step before each tool call reduces failures
THE THREE-WEEK PLAN (what enthusiasm proposes)
build the review step into the agent loop
design a prompt for it, tune the prompt
add a retry path for calls the reviewer rejects
run the full 400-task suite, three seeds, with and without
write it up
-> three weeks before the first number
THE TWO-HOUR PLAN (what to do instead)
take 200 tool calls already in the logs, of which 31 failed
show each one to the model, cold, with the tool schema, and ask
a single question: will this call succeed, yes or no
count how often it catches the ones that failed
THE POINT: if the model cannot tell a failing call from a working
one when handed it in isolation, no amount of loop engineering
will make it do so in flight. The whole idea rests on that one
ability, so measure that one ability.
WHAT IT RETURNED
caught 6 of the 31 real failures (19% recall)
flagged 44 of the 169 working calls (26% false positive rate)
-> a reviewer that rejects a quarter of good calls to catch a
fifth of bad ones. The retry path would cost more than the
failures do.
DECISION dropped, after two hours, with a number rather than a
feeling. The write-up is four lines long and it is the
most useful thing on the problem list this month.Notice what the two-hour version skipped: the loop, the retry path, the prompt tuning, the full suite. All of those are engineering that would have been real work, and none of them could have rescued a reviewer that cannot distinguish a good call from a bad one when the question is handed to it directly. That is the general recipe. Find the one assumption the whole direction rests on, strip away everything that is not that assumption, and measure it on data you already have.
The objection that always arises is that the crude version is unfair to the idea, and sometimes it is. The answer is not to skip it but to say in advance what the crude version has to show for the careful one to be worth building. If you cannot name that number beforehand, the crude experiment was never going to change your mind and you should admit that before running it rather than afterwards.
Write down what would change your mind#
Decide in advance what result would make you stop. Written before you see data, it is a genuine commitment. Decided afterwards, you will find a reason the number does not count, and you will be sincere about it.
It takes ten minutes and it fits on a screen. The value is entirely in the timestamp: this is a promise made by someone who is not yet invested, to the person who will be invested in six weeks and will not be able to see straight.
PRE-REGISTRATION, 2 May, before any runs
CLAIM reranking the retrieved passages reduces answer errors
QUANTITY exact-match accuracy on our 400 held-out questions
COMPARISON same pipeline, reranker removed. Same retriever, same
top-k into the reader, same prompt.
NOISE FLOOR measured first, before the treatment: three seeds of
the baseline gave 71.2 / 72.0 / 72.6, so a spread of
1.4 points. Anything under that is nothing.
I STOP IF the gain is under 1.4 points averaged over three seeds
I CONTINUE IF the gain is over 3 points, and then the next thing I
do is the latency measurement, not more accuracy work
IN BETWEEN 1.4 to 3 points: run three more seeds, once. If it is
still in between, stop. An effect I need six seeds to
see is not an effect anyone will benefit from.
WHAT I EXPECT about 2 points. I am writing this down so that I
cannot later claim I predicted whatever happened.
NOT ALLOWED changing the eval set, changing the metric, dropping
the seeds I dislike, or deciding the held-out set was
unrepresentative after seeing the numbers.Measuring the noise floor before the treatment is the non-obvious discipline in that document, and skipping it is how most pre-registrations become decorative. If you set the stopping threshold after you know the spread, you will set it just below whatever you got. Measure the baseline variance first, write the threshold from it, and you have bound yourself with a number that came from the world rather than from your hopes.
Sunk cost#
Months invested is not evidence
The most expensive research mistake is continuing because of what has already been spent. The question is always whether the next month is the best use of the next month, given what you now know. What it cost to get here is not part of that calculation.
This is easy to agree with and hard to apply, because sunk cost never presents itself as sunk cost. It arrives as a technical argument. Here is the same situation reasoned both ways, five months into a direction that has not produced a positive result.
THE ARGUMENT THAT FEELS LIKE ANALYSIS
"We are five months in and the infrastructure is finally working.
The last three failures were all setup problems. It would be
mad to walk away now, right when we can finally run things
properly - the real experiment has not even started."
Every clause is true. None of it is evidence about whether the
hypothesis is correct. Read it again with that in mind: it argues
entirely from cost and readiness, never from a result.
THE FORWARD CALCULATION
What do I now believe, that I did not believe in December?
- the effect, if it exists, is smaller than 2 points, because
we would have seen a larger one through the setup noise
- the setting where it should be strongest is the one where we
measured least, and that was not an accident: it is hardest
What would the next month buy?
- one clean run of the experiment we have wanted since December
- probability I would put on a positive result: maybe 15%
What else could the next month buy?
- three of the afternoon-sized questions on the list, two of
which change a default in a system we ship
DECISION one clean run, timeboxed to three weeks, because the
experiment is now cheap and an unanswered question is worth
closing. Then stop regardless of the outcome's ambiguity. The
five months are not in this calculation anywhere.The distinction that survives is this one: what you have already spent is irrelevant, but what you have already built is an asset, and an asset changes the forward cost. Five months of belief is worth nothing. Five months of working infrastructure genuinely makes the next experiment cheaper, and cheapness is a legitimate reason to run it. The failure is using the first to argue for the second.
When to persist#
Not every negative result is a verdict. Persist when the failure is clearly mechanical rather than conceptual, when you can name specifically what went wrong, or when the idea keeps being independently suggested by people you respect. Abandon when you are persisting because stopping would feel like failure.
The distinction is sharper than it sounds, and the test is whether you can name the fault before you fix it. Two cases that look identical from the outside — a flat result, an enthusiastic researcher — and separate cleanly under that test.
CASE 1 MECHANICAL. Persist.
Result fine-tuning made no difference at all. Exactly none:
the tuned model's outputs are identical to the base
model's on 400 of 400 examples.
Diagnostic "identical on all 400" is not what a real null result
looks like. A method that does not help still moves
outputs around.
Named fault I can state it before fixing it: the adapter weights
are probably not being loaded at inference.
Cost twenty minutes to check
Verdict this is a bug, not a finding. Fix it, then re-judge.
CASE 2 CONCEPTUAL. Stop.
Result the method helps by 0.4 points, seed spread 1.1, on
the fourth variant of the setup we have tried
Diagnostic each variant was chosen after the previous one failed,
and each was justified afterwards. Nothing was
pre-registered after the first one.
Named fault none. The explanation for each failure was different
and arrived after the number did.
Verdict stop. Four post-hoc explanations is not four pieces
of evidence, it is one piece of evidence about me.
THE TEST can you name the fault BEFORE you fix it, and does the
fix have an independent reason to be right? One yes is persistence.
Two nos is a search for a setting where the noise is kind.There is a third case where persistence is right and neither test applies: the idea keeps being independently proposed by people whose judgement you rate, and nobody has actually tested it. That is weak evidence about the hypothesis, but it is strong evidence that the answer is wanted, which means a clean negative result has a reader. Under those conditions the experiment is worth finishing even when you expect it to fail, and you should say that out loud in advance so the failure is not a disappointment when it arrives.
Where people go wrong
Building the careful version before the crude one. If the crude version shows nothing, the careful one almost never rescues it, and you have spent a month finding that out.
If this comes up in an interview
How do you decide whether to abandon a research direction?
Design the cheapest experiment that could falsify it and run that first, and write down in advance what result would make me stop. Then judge whether the next month is the best use of the next month, ignoring what has already been spent.
When is persisting after a negative result right?
When the failure is mechanical rather than conceptual and I can name exactly what went wrong, or when strong people keep independently suggesting the idea. Not when I am persisting because stopping would feel like failure.
Then they push further
How do you tell those apart in the moment?
Whether I can name the fault before I fix it, and whether the fix has an independent reason to be right. A tuned model producing outputs identical to the base model on every example is a loading bug and I can say so before checking. A fourth variant of the setup, each one chosen after the last failed and justified afterwards, is not four pieces of evidence about the idea.
What would you pre-register before running an experiment, and why bother?
The quantity, the comparison, the noise floor measured on the baseline first, and the threshold that makes me stop — plus an explicit list of the moves I am not allowed to make afterwards, such as changing the eval set or dropping seeds. The point is the timestamp: it is a promise made by someone not yet invested to the person who will be invested in six weeks and will find sincere reasons why the disappointing number does not count.
Build this: Take a direction you are interested in. Design the cheapest experiment that could kill it, write down what result would stop you, then run it.
Watch it explained
This is one section of Research taste, 18 hours across 5 sections. Read it in full to get the whole topic in order.