Asked about Probability, and reading an experiment honestly
Your new prompt scores 86 percent against 82 percent on a 50-case eval. Ship it?
An answer that works
Not on that evidence. The margin of error on 50 samples is around ten points, so the difference is inside the noise. Two identical systems scored on 50 cases each differ by four points or more about 62 per cent of the time. Look at which specific cases changed, and if the direction seems real, enlarge the eval set before claiming an improvement.
Then they push further
Both were scored on the same 50 cases. Does that change your answer?
It makes the comparison much stronger, so analyse it that way: ignore the totals and count only the cases where the two disagree. If they disagree on five and the new prompt wins three, that is a coin flip. With five disagreements even a clean sweep gives a two-sided p of 0.0625, so nothing this eval could have returned would clear 0.05.
How many cases would you need?
Ask for disagreements rather than cases. Six disagreements all falling the same way is the smallest result that reaches significance in a sign test. For an unpaired comparison, the margin of error shrinks with the square root of n, so separating a genuine four-point difference needs on the order of a thousand cases per side, which is why paired comparison is worth the trouble.
What gets marked up
A passable answer says the sample is too small. A strong one gives a number for how often noise alone produces that gap, and moves to the paired analysis without being prompted.
Reading an answer is not being able to give one. The section that teaches this is Probability, and reading an experiment honestly, 6 hours of Probability & statistics.
Learn this properlyAlso asked about Probability & statistics