Asked about Is the difference real? Statistics for evals
Your change moved the eval from 80% to 83% on 60 cases. Do you ship it?
An answer that works
Not on that evidence. The interval on 60 cases is roughly plus or minus ten points, so the difference is well inside noise. I would look at the cases where the versions disagree, check run-to-run variance by rerunning the baseline, and grow the set if the decision matters.
Then they push further
How would you compare the two versions properly?
Run both on the same cases and look only at the ones where they disagree. If B wins most of a reasonable number of disagreements, that is evidence. A paired test such as McNemar's on those counts gives a p-value.
The baseline itself scores 78% then 82% on two identical runs. What now?
Then single runs cannot distinguish changes smaller than about that spread. Average several runs per version, lower sampling randomness where the product allows, and treat anything inside the observed spread as no change.
Leadership wants a number for the launch review. What do you give them?
The pass rate with its interval and sample size, the failure modes that remain with their counts, and the cost of the worst one. A single percentage invites a false sense of precision.
What gets marked up
Strong candidates immediately question sample size and variance, and know to compare on paired cases. Very strong ones mention rerunning the baseline. Weak answers accept a three point gain at face value.
Reading an answer is not being able to give one. The section that teaches this is Is the difference real? Statistics for evals, 4 hours of Evaluation.
Learn this properlyAlso asked about Evaluation