Asked about Isolating what actually caused the improvement
How do you know your improvement is real?
An answer that works
A baseline tuned as hard as the method, one variable changed at a time, ablations showing each component earns its place, and several seeds with variance reported. A difference smaller than seed-to-seed variance is not a result.
Reading an answer is not being able to give one. The section that teaches this is Isolating what actually caused the improvement, 5 hours of Experiment design.
Learn this properlyAlso asked about Experiment design
What is an ablation study and why does it matter?How many seeds should you run?What is a confound, and how do you control for it?Why decide your metric before running an experiment?Your new method beats the baseline by 0.8 points on one run. How do you decide if it is real?Your method has three components. How do you show which ones matter?