Asked about Statistics for comparing training runs
Your new method beats the baseline by 0.8 points on one run. How do you decide if it is real?
An answer that works
Run both with several seeds to measure training variance, compute a paired bootstrap interval over the test examples, and check the gap is clearly larger than both sources of noise. If I tried many variants to find this one, correct for that or confirm on fresh data.
Then they push further
You only have budget for three seeds each. What then?
Report means and spreads honestly, use the paired test-set bootstrap, and describe the result as preliminary if the gap is within seed variation. Spend remaining budget on the comparison that matters most rather than on more variants.
Why is the baseline so often the weak point?
It usually gets less tuning than the new method. A fair comparison gives the baseline the same tuning budget; many published gains shrink when that is done.
What gets marked up
Strong answers separate seed variance from test-set variance, use paired comparisons, and mention multiple comparisons and baseline tuning.
Reading an answer is not being able to give one. The section that teaches this is Statistics for comparing training runs, 5 hours of Experiment design.
Learn this properlyAlso asked about Experiment design