Revising, reviewing and responding
By the end: You can improve a draft systematically and handle review well.
- Revising for clarity
- Cutting
- Getting useful feedback
- Responding to reviewers
New to writing & communication? Turning a result into something someone else can use — a paper, an internal report, a design document. Concretely it is four decisions: what the single claim is, in what order the evidence has to arrive for a reader to accept it, which figures carry the argument without their caption, and what you did not show, said plainly rather than left for the reader to notice. Why it matters
A first draft is not the work, it is the raw material. This section is about what turns one into something people can follow: revising in passes rather than trying to fix everything at once, cutting the third of the draft that is hedging and restatement, and asking for feedback in a form that produces something usable — have someone state your main point in one sentence and see whether it is the one you meant. It also covers replying to review without either capitulating or arguing.
Revise in passes#
- 1.Structure: is the argument in the right order, and is the contribution clear from the first page?
- 2.Paragraphs: does each one make a single point?
- 3.Sentences: cut every word that does not carry meaning.
- 4.Figures: does each one make its point without the text?
Doing them in separate passes is not fussiness. Structure and sentences are different jobs and they interfere: an hour spent polishing the wording of a paragraph makes you unwilling to delete it, and the structural pass is mostly deletion. Fix the order first, while everything is still cheap to throw away.
One paragraph, before and after#
Advice about clarity is easy to agree with and hard to apply, so here is a paragraph of the kind that appears in real first drafts, followed by the same content fixed, followed by a list of exactly what changed. Read the first one at the speed you would read a paper you did not choose.
BEFORE
In this work, we propose a novel approach for the chunking of documents
in the context of retrieval-augmented generation systems. It is well
known that the way in which documents are split into chunks can have a
significant impact on the overall quality of the resulting system, and a
number of different strategies have been proposed in the literature to
address this issue. In order to investigate this further, we performed a
comprehensive set of experiments in which we evaluated a variety of
different chunking strategies across several different document
collections. Our results demonstrate that our proposed approach is able
to achieve improved performance relative to the baselines that we
considered, which we believe suggests that chunk boundaries may be an
important consideration for practitioners.AFTER
Splitting documents on their section boundaries retrieves better than
splitting them every 512 tokens. We compared six chunking strategies on
four document collections, holding the retriever and the generator
fixed; boundary-aware chunking scored 8.4 points higher on answer
accuracy, averaged over five seeds. The gain was largest on long
structured documents and absent on short unstructured ones.Seven things changed, and it is worth naming them separately because they are separate habits. One: the result moved to the first sentence, so the reader knows the finding before the method. Two: a novel approach became the thing itself, splitting on section boundaries, which is both shorter and checkable. Three: a significant impact became 8.4 points. Four: the stack of hedges at the end — we believe, suggests, may be — became a statement, with the honesty moved into a condition instead. Five: the quantity words, comprehensive, a variety of, several different, became quantities: six, four, five. Six: three phrases of throat-clearing were deleted outright. Seven: a limitation was added, which makes it longer and better.
Most of that is mechanical enough to count. The checks below are crude on purpose — a fixed list of stock phrases and hedges, the word count, and how many digits the paragraph contains — and crude is the point, because a draft that scores badly here is badly written for reasons the author can see without an argument about taste.
BEFORE = """In this work, we propose a novel approach for the chunking of documents
in the context of retrieval-augmented generation systems. It is well known that the
way in which documents are split into chunks can have a significant impact on the
overall quality of the resulting system, and a number of different strategies have
been proposed in the literature to address this issue. In order to investigate this
further, we performed a comprehensive set of experiments in which we evaluated a
variety of different chunking strategies across several different document
collections. Our results demonstrate that our proposed approach is able to achieve
improved performance relative to the baselines that we considered, which we believe
suggests that chunk boundaries may be an important consideration for practitioners."""
AFTER = """Splitting documents on their section boundaries retrieves better than
splitting them every 512 tokens. We compared six chunking strategies on four document
collections, holding the retriever and the generator fixed; boundary-aware chunking
scored 8.4 points higher on answer accuracy, averaged over five seeds. The gain was
largest on long structured documents and absent on short unstructured ones."""
HEDGES = ["may", "might", "could", "suggests", "we believe", "potentially",
"somewhat", "relatively", "arguably", "appears to", "tends to"]
FILLER = ["in this work", "it is well known that", "in order to", "the way in which",
"in the context of", "a number of", "a variety of", "comprehensive",
"novel", "several different", "is able to", "the resulting",
"overall", "in the literature", "to address this issue"]
def report(name, text):
low = " ".join(text.split()).lower()
words = len(low.split())
digits = sum(c.isdigit() for c in low)
h = [p for p in HEDGES if p in low]
f = [p for p in FILLER if p in low]
print("%-6s words %3d digits %2d hedges %d %s" % (name, words, digits, len(h), h))
for i in range(0, len(f), 3):
print(" filler:" if i == 0 else " ", ", ".join(f[i:i + 3]))
report("before", BEFORE)
report("after", AFTER)
# before words 125 digits 0 hedges 3 ['may', 'suggests', 'we believe']
# filler: in this work, it is well known that, in order to
# the way in which, in the context of, a number of
# a variety of, comprehensive, novel
# several different, is able to, the resulting
# overall, in the literature, to address this issue
# after words 57 digits 5 hedges 0 []Cut#
First drafts are almost always too long. Hedges, restatements and throat-clearing accumulate. Cutting a third of a draft usually makes it clearer, not thinner.
The third that goes is not evenly distributed, and knowing where it hides makes the pass fast. These replacements are close to mechanical, and doing them in one sweep before you think about anything else usually removes a fifth of the text.
what you wrote what it becomes
It is important to note that X X
In order to Y To Y
Due to the fact that Z Because Z
We performed experiments in which we We compared
evaluated
has a significant impact on changes it by <number>
demonstrates improved performance scored <number> higher
a number of / a variety of six
It is well known that X. [X is your delete the sentence
own field's common ground]
As mentioned in Section 3, ... delete; they can read Section 3
In this paper, we ... [in the body] delete; they know where they areThree things are worth protecting from the cutting pass, because they look like padding and are not. The sentence that says what the result does not show. The sentence that dispatches the obvious alternative a reader is currently thinking of. And the definition of a term you are using in a slightly non-standard way. Each of those is a sentence that prevents a misreading, and the cost of a misreading is a reviewer who stops believing you.
Useful feedback#
Ask a specific question rather than for general comments. Ask someone to state your main contribution in one sentence after reading it. If they cannot, or state something else, the draft is not doing its job regardless of how the prose reads.
Any question of the form what do you think invites the reader to look for wording problems, because wording is what is easy to comment on. These five are worth asking instead, and each one fails loudly rather than quietly.
- 1.In one sentence, what did this paper find? If the answer is not your main claim, the abstract has failed, and nothing downstream of it will be read charitably.
- 2.Which claim did you least believe? This gets you the weakest link rather than the most visible one, and it is the claim a reviewer will attack.
- 3.Where did you stop and re-read? Do not ask them to explain why. The location is the finding; the diagnosis is your job.
- 4.What did you expect to be in here that was not? This catches the missing baseline and the missing ablation, which are the two rejections you cannot fix after submission.
- 5.If you had to cut a page, which page? A reader who can answer this instantly has told you which section is not carrying its weight.
Ask different questions of different readers. Someone inside your area should be asked question two and question four, because those need the literature. Someone outside it should be asked question one and question three, because they will get confused where a real reader gets confused, and an expert will silently repair your sentences as they read.
Responding to review#
A review is a list of things that at least one careful reader thought after reading your draft, and a response is a document with one job: to let a busy editor see, without re-reading the paper, what changed. Here is a real-shaped review comment and two responses, because the interesting case is not the one where the reviewer is right.
REVIEWER 2
The paper claims boundary-aware chunking is better, but Table 2 shows the
improvement is within the standard deviation on two of the four collections.
The authors should run more seeds. Additionally, the comparison to
fixed-size chunking is unfair, since the fixed-size baseline uses no
overlap, which is not standard practice.RESPONSE TO R2.2 (the overlap baseline) -- we agree
You are right, and we have changed it. We reran fixed-size chunking with
20% overlap, the default in both libraries we cite, on all four
collections. The gap narrows from 8.4 to 5.9 points and the ordering is
unchanged. Table 2 and the abstract now report the overlapping baseline;
Appendix C keeps the original numbers. This was the correct baseline and
we should have used it.RESPONSE TO R2.1 (two collections inside the standard deviation) -- we disagree
We agree with your reading of Table 2 and have narrowed the claim: the
abstract now says the gain is largest on long structured documents and is
not distinguishable from zero on the two short unstructured collections
(Section 5.3, revised).
We have not run more seeds, and we would rather say why than quietly not
do it. Our five seeds vary only the chunker's tie-breaking; the
collections and the retriever are fixed. More seeds would therefore
narrow the interval around the same point estimate, and would not address
the question you are raising, which is whether the effect exists on short
documents at all. The experiment that would address it is a fifth
collection of short unstructured documents, which we did not have. We
have added this to the limitations (Section 7) and named it as the
experiment that would settle the question.Notice what the second response does not do. It does not argue that the reviewer misread the table, and it does not comply in order to be agreeable — running more seeds would have cost a week and produced a tighter interval around a result that was never in dispute. Conceding the claim while declining the method is a stronger position than either, and it is available far more often than people use it, because it rests on knowing exactly what your own experiment varied.
Three failures lose a reviewer outright, in descending order of how often they happen. Agreeing warmly and changing nothing, which is visible immediately and reads as bad faith. Answering a question adjacent to the one asked, which reads as evasion whether or not it was. And arguing about whether the reviewer should have been confused, which is the one that feels most justified at the time and is never worth it: their confusion is data about the text, and it is the only data of that kind you will get before publication.
Assume the reviewer is a careful reader who got confused
When a reviewer misunderstands, the draft usually allowed the misunderstanding. Fixing the text so the next reader does not make the same mistake is more productive than arguing that the reviewer was wrong.
Where people go wrong
Asking for general comments. You get vague praise or scattered edits, and not the answer to whether the contribution is clear.
If this comes up in an interview
How do you get useful feedback on a draft?
Ask a specific question, and ask the reader to state the main contribution in one sentence. If they cannot, the draft is not communicating it, whatever the quality of the prose.
How do you handle a reviewer who misunderstood your work?
Treat it as evidence the draft allowed the misunderstanding and revise the text so the next reader does not repeat it. Arguing the reviewer was wrong rarely helps.
A reviewer asks for an experiment you think is the wrong experiment. What do you write?
Accept the observation behind the request before addressing the request, because the observation is usually correct even when the proposed fix is not. Then narrow whatever claim prompted it, decline the specific experiment with a technical reason — what your existing runs varied, and why more of them would not answer the question being raised — and name the experiment that would settle it, adding it to the limitations. Conceding the claim while declining the method is a stronger position than either capitulating or arguing, and it rests on knowing exactly what your own experiment varied.
Then they push further
What are the ways a response loses a reviewer?
Agreeing warmly and changing nothing, which is visible immediately and reads as bad faith. Answering an adjacent question, which reads as evasion. And arguing about whether they should have been confused, which is the most tempting and never worth it: their confusion is data about the text and it is the only data of that kind you get before publication.
What gets marked up. Marked up for separating the observation from the requested remedy, and for naming what would settle it. Candidates who either fold on every point or defend every point both score badly.
Show me how you would tighten a paragraph.
Work in one pass over mechanical things before touching anything else: delete the throat-clearing openers, replace quantity words with quantities, cut stacked hedges down to one statement plus an explicit condition, and move the result to the first sentence. A useful crude check is how many digits the paragraph contains — an abstract with none is usually an abstract with no evidence in it. Protect three things from the cut: the sentence saying what the result does not show, the sentence dispatching the obvious alternative, and any definition of a term you are using non-standardly.
Build this: Take a draft, cut it by a third, and ask someone to state its main contribution before and after. Compare the answers.
Watch it explained
This is one section of Writing & communication, 16 hours across 4 sections. Read it in full to get the whole topic in order.