Asked about Hypothesis tests and intervals, practically
Your dashboard has twenty metrics and one moved with p = 0.03. What do you do?
An answer that works
Treat it as nothing. With twenty independent metrics at a 5 per cent threshold, the chance at least one fires when nothing has changed is one minus 0.95 to the twentieth, about 64 per cent, and the average number of false winners is exactly one. Name the primary metric before the experiment and treat everything else as a hypothesis for the next one.
Reading an answer is not being able to give one. The section that teaches this is Hypothesis tests and intervals, practically, 4 hours of Probability & statistics.
Learn this properlyAlso asked about Probability & statistics
Your new prompt scores 86 percent against 82 percent on a 50-case eval. Ship it?A test for a condition affecting 1 in 1000 people is 99 percent accurate. You test positive. What is the chance you have it?Why is accuracy misleading on imbalanced data?Why report p95 latency rather than the mean?What is maximum likelihood estimation?Why is cross-entropy the loss for classification?How would you put a confidence interval on an eval score?What does it mean for a model to be calibrated?
All 20 questions on Probability & statistics