Experiment design interview questions
7 questions across 4 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Isolating what actually caused the improvement
Read the section that teaches thisIf this comes up in an interview
How do you know your improvement is real?
A baseline tuned as hard as the method, one variable changed at a time, ablations showing each component earns its place, and several seeds with variance reported. A difference smaller than seed-to-seed variance is not a result.
What is an ablation study and why does it matter?
Removing one component at a time and measuring the loss, to show each part contributes. It often reveals that something assumed important does not help, which is better to find yourself than in review.
How many seeds should you run?
Enough to know the variance, usually three to five for expensive runs and more when cheap. Without knowing the spread you cannot say whether a difference means anything.
Controls, confounds and pre-registration
Read the section that teaches thisIf this comes up in an interview
What is a confound, and how do you control for it?
Something that changed alongside the variable you meant to test, such as extra compute or longer training. Control it by matching the baseline on everything except the one change, including parameter count, data and training time.
Why decide your metric before running an experiment?
With many metrics available, one will improve by chance. Committing beforehand turns the result into a genuine test rather than a search for something that looks good.
Statistics for comparing training runs
Read the section that teaches thisIf this comes up in an interview
Your new method beats the baseline by 0.8 points on one run. How do you decide if it is real?
Run both with several seeds to measure training variance, compute a paired bootstrap interval over the test examples, and check the gap is clearly larger than both sources of noise. If I tried many variants to find this one, correct for that or confirm on fresh data.
Then they push further
You only have budget for three seeds each. What then?
Report means and spreads honestly, use the paired test-set bootstrap, and describe the result as preliminary if the gap is within seed variation. Spend remaining budget on the comparison that matters most rather than on more variants.
Why is the baseline so often the weak point?
It usually gets less tuning than the new method. A fair comparison gives the baseline the same tuning budget; many published gains shrink when that is done.
What gets marked up. Strong answers separate seed variance from test-set variance, use paired comparisons, and mention multiple comparisons and baseline tuning.
Designing an ablation study, worked through
Read the section that teaches thisIf this comes up in an interview
Your method has three components. How do you show which ones matter?
Run the baseline plus each component alone, and the full method minus each component, with several seeds each. Compare effects against seed variation, and check interactions for the components that matter with a small grid.
Then they push further
Removing component A costs 2 points but adding it alone to the baseline gains nothing. What does that mean?
A interacts with another component: it only helps in combination. The paper should describe it that way rather than claiming A is independently useful.
What gets marked up. Marked up for both add-one and remove-one directions, seeds, and recognising interactions.
These are the questions for one topic. The whole question bank covers every topic on the site.