Designing an ablation study, worked through

An ablation study answers one question for each part of your method: if this were removed, would the result change? It is the difference between a method and a pile of things that happened to be in the code when the good number appeared. This section runs a real one — five components of an ordinary text classifier, each removed in turn — and the outcome is the usual one, which is that most of them earn nothing and one is actively harmful.

Remove one thing at a time#

The method here is a TF-IDF text classifier over four newsgroups, with five deliberate choices: lowercasing, stopword removal, bigrams, sublinear term frequency, and dropping terms that appear in only one document. Each is the kind of thing that ends up in a method section as though it were load-bearing.

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

cats = ["sci.med", "sci.space", "rec.autos", "talk.politics.guns"]
data = fetch_20newsgroups(subset="train", categories=cats,
                          remove=("headers", "footers", "quotes"), random_state=0)

def score(**override):
    opts = dict(lowercase=True, stop_words="english", ngram_range=(1, 2),
                sublinear_tf=True, min_df=2)
    opts.update(override)                       # one change per call
    pipe = make_pipeline(TfidfVectorizer(**opts), LogisticRegression(max_iter=2000))
    return cross_val_score(pipe, data.data, data.target, cv=5).mean()

full = score()
print(f"full                   {full:.4f}")
for label, change in [("no bigrams",          dict(ngram_range=(1, 1))),
                      ("no stopword removal", dict(stop_words=None)),
                      ("no lowercasing",      dict(lowercase=False)),
                      ("no sublinear tf",     dict(sublinear_tf=False)),
                      ("no min_df filter",    dict(min_df=1))]:
    s = score(**change)
    print(f"{label:22} {s:.4f}   {s - full:+.4f}")

# full                   0.8883
# no bigrams             0.8861   -0.0022
# no stopword removal    0.8754   -0.0129
# no lowercasing         0.8840   -0.0043
# no sublinear tf        0.8887   +0.0004
# no min_df filter       0.8956   +0.0073

This is the ordinary result of an honest ablation, and it is why the exercise is worth doing on your own work before a reviewer does it on your behalf. Of five components, one matters (stopword removal, 1.3 points), two are marginal, one is inert, and one is harmful. The method should now lose two of its five parts, which makes it simpler to describe, faster to run and easier for someone else to reproduce.

Interactions: why one at a time is not enough#

Removing one thing at a time is necessary and not sufficient, because components can cover for each other. Take the two most interesting rows above and remove both together. Individually they sum to a loss of 0.56 points. Together they lose 1.50.

both = score(stop_words=None, min_df=1)
singles = (score(stop_words=None) - full) + (score(min_df=1) - full)
print(f"both removed  {both:.4f}   {both - full:+.4f}")
print(f"sum of singles                {singles:+.4f}")

# both removed  0.8732   -0.0150
# sum of singles                -0.0056

So the honest conclusion is not the one the single-ablation table suggested. The min_df filter is not useless — it is redundant with stopword removal on this data, and which of the two you keep is a choice rather than a finding. A table of single ablations alone would have reported the wrong thing confidently, which is why the pairs you suspect of interacting are worth the extra runs.

Budgeting the compute#

Every pair is not affordable: n components give n single ablations, and n times n minus one over two pairs — five components is five and ten, twenty-five components is twenty-five and three hundred. So spend the budget deliberately rather than exhaustively.

• Every component you claim credit for gets a single ablation. This is not optional; it is the table.
• Pairs only where you have a reason to suspect interaction — two things addressing the same problem, as above.
• Ablate at the smallest scale where the effect is still visible, then confirm the two or three that matter at full scale.
• Run the ablations as you build, not at the end. A component ablated the week it was added costs one run; ablated at the end it costs a rerun of everything.
• Seeds before extra ablations: three seeds on five components beats one seed on fifteen, because a single-seed delta smaller than the seed spread is not a measurement.

What reviewers will ask#

question                                  what it is really checking
"Did you ablate X?"                       whether X earns its place
"What is the variance on these deltas?"   whether the deltas exist at all
"Does the ordering hold on dataset B?"    whether this is a method or a dataset
"What happens if you remove two?"         whether the components are redundant
"Why is component Y in the method         whether you read your own table
  if removing it costs 0.0004?"
"What would have made you drop this?"     whether the study could have failed