Writing a one-page research pitch

A research pitch is one page arguing that a question is worth someone's time. It is the artefact that gets a direction funded, staffed or killed, and writing one is the fastest way to discover that an idea you have been carrying for a month does not survive contact with a page. This section gives the questions a pitch has to answer, then writes one out in full, because the advice to be clear about your contribution is useless and the paragraph that does it is not.

The questions a pitch must answer#

Five, in this order. A reader who cannot find all five in under two minutes will decline, and usually without telling you which one was missing.

1. what is the question          stated so it could be answered yes or no
2. why does it matter            what changes if the answer is yes
3. why is it open                why the obvious approaches have not settled it
4. what is the first experiment  small, specific, and doable in weeks
5. what would change your mind   the result that would make you stop

One written out#

Here is the whole thing, at the length it should be. The subject is deliberately ordinary — it is not a famous open problem, and the pitch is stronger for that.

Does retrieval quality degrade when a model is fine-tuned on its own
retrieved outputs?
-------------------------------------------------------------------

THE QUESTION.  Systems that fine-tune on their own logged outputs are now
common, and those outputs were produced with retrieved context. If the
model learns to rely on information that was in the context rather than
in its weights, its behaviour without that context should get worse. Does
it, and by how much?

WHY IT MATTERS.  If it does, the standard loop of logging production
traffic and fine-tuning on it quietly degrades the model for every path
that does not retrieve -- which for most products is the majority of
calls. Nobody would see it, because the eval set is drawn from the same
retrieved-context distribution.

WHY IT IS OPEN.  The obvious check is to evaluate with and without
context, and papers on self-training do not report the second number. The
closest work measures hallucination after self-training but always with
retrieval present. So the effect could be large, zero, or reversed, and I
cannot find the experiment.

FIRST EXPERIMENT.  Two weeks, one GPU. Take a 7B instruct model, generate
answers to 20,000 questions with retrieval, fine-tune on them, and
evaluate the tuned and untuned models on held-out questions BOTH with and
without context. The quantity of interest is the change in the no-context
score. Baseline: the same fine-tune on the same answers with the context
stripped from the prompt.

WHAT WOULD CHANGE MY MIND.  If the no-context score moves by less than
the seed-to-seed spread (about a point on this setup, measured over three
seeds), there is no effect worth chasing and I stop. If it drops by more
than three points I will run the ablation on context length, since the
mechanism predicts a larger effect with more retrieved text.

RISKS.  The effect may be specific to one model family; I will confirm on
a second before writing anything up.

Read the third paragraph again, because it is the one people write badly. It does not claim nobody has thought about this; it says precisely what the nearest work measured and what it did not, which is a checkable statement rather than a boast. A reader who knows the literature can correct you in one line, and that correction is worth more than the month you would otherwise spend.

The smallest experiment first#

The first experiment in a pitch should be embarrassingly small — small enough that you would run it this fortnight rather than after a planning cycle. Its job is not to answer the question but to establish that the effect exists and is measurable at all. Most pitches die here for a good reason: the author cannot name an experiment under a month, which usually means the question is not yet sharp enough to test.

• Prefer the smallest model and dataset that could show the effect. If it only appears at scale, say so and explain how you know.
• Name the quantity you will measure, not the thing you will build.
• State the baseline in the same sentence as the treatment, or the result will be uninterpretable.
• Give the number the result has to beat, which means knowing your noise floor before you start.
• Cost it in days and GPUs. A pitch without a number here reads as wishful.

What result would change your mind#

Write this before running anything, and write it as a number rather than a direction. 'If it gets worse I will reconsider' commits you to nothing; 'if the no-context score moves by less than a point, which is the seed spread here, I stop' commits you to something you cannot argue your way out of at the time. That is the entire function of the sentence: it is a promise made by someone who is not yet invested to someone who will be.

Getting useful criticism#

Send the page to two people, and ask each a specific question rather than for thoughts. Ask someone who knows the area whether the third paragraph is true — whether the work you claim does not exist really does not exist. Ask someone who does not know the area whether they can restate your question back to you, because if they cannot, the first paragraph has failed and nothing after it will be read.

The useful response to either is not defence. If the literature reader names a paper that already did it, that is the pitch working: it cost you an afternoon instead of a quarter. And a direction that survives both readings is one you can start on knowing that the fastest ways of being wrong have already been tried.