Asked about Controls, confounds and pre-registration
Why decide your metric before running an experiment?
An answer that works
With many metrics available, one will improve by chance. Committing beforehand turns the result into a genuine test rather than a search for something that looks good.
Reading an answer is not being able to give one. The section that teaches this is Controls, confounds and pre-registration, 4 hours of Experiment design.
Learn this properlyAlso asked about Experiment design
How do you know your improvement is real?What is an ablation study and why does it matter?How many seeds should you run?What is a confound, and how do you control for it?Your new method beats the baseline by 0.8 points on one run. How do you decide if it is real?Your method has three components. How do you show which ones matter?