Asked about Building the dataset
You have a bug that affects about 6% of traffic and a fix for it. How do you show the fix works?
An answer that works
Not with the general eval set. At 6% prevalence, a hundred random cases contain about six that exercise the bug, and a paired test on that few detects a working fix less than a tenth of the time. I would pull twenty to forty cases from the reports of that specific bug and use those to show it is fixed, then use the general set to check nothing else regressed and to say how much it mattered overall. Two sets, two questions.
Then they push further
Does the targeted set not overstate the improvement?
Yes, and deliberately. Every case in it was selected for being broken, so its pass rate cannot be read as a product metric. That is why it is reported separately from the traffic-shaped set rather than merged into it.
How do you sample the general set so rare categories are usable?
Stratify with a floor of twenty or thirty per category, then weight the categories back to the real traffic mix when reporting the headline number. Without the floor the rare categories land with two or three cases and their pass rate can only be 0, 50 or 100 per cent.
What gets marked up
Strong answers separate the question 'did this fix work' from 'how much does this matter', and use a different set for each. Weak answers propose adding a few cases to the existing set and re-running it.
Reading an answer is not being able to give one. The section that teaches this is Building the dataset, 5 hours of Evaluation.
Learn this properlyAlso asked about Evaluation