Data quality & labelling interview questions
4 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Where ground truth comes from
Read the section that teaches thisIf this comes up in an interview
Two annotators disagree on 30 percent of labels. What does that tell you?
That the task definition is ambiguous, not that someone is careless. It also caps achievable model performance at around that agreement rate, since there is no stable target above it. The fix is clearer guidelines, starting from the disagreements.
Then they push further
Another team reports 95 per cent agreement. Is their task better defined?
Not necessarily, and on a rare-class task almost certainly not. Two annotators who ignore the item entirely and guess at the prevalence agree 95 per cent of the time when the positive class is 2 per cent, because they both say no to nearly everything. Ask for Cohen's kappa and the class prevalence; kappa subtracts the agreement expected from the marginals, so chance agreement scores zero.
You rewrote the guidelines and kappa did not move. What now?
That usually means the disagreement was genuine ambiguity rather than unclear wording, so the label set is wrong rather than the prose. If two careful readers keep splitting on the same kind of item, that kind of item is a third category, and adding it, or adding an explicit not-applicable, recovers more agreement than further clarification.
What gets marked up. Marked up for treating disagreement as a definition problem, for knowing raw agreement is inflated by class imbalance, and for naming kappa.
How do you write labelling guidelines?
Define each label with concrete positive and negative examples rather than adjectives, state explicitly how to handle ambiguity, include a not-applicable option, then have someone else label with them and fix whatever they got wrong.
Finding and fixing bad data
Read the section that teaches thisIf this comes up in an interview
You suspect many labels in your training set are wrong. How do you find them?
Get out-of-fold predictions for every example, rank by how confidently the model disagrees with the given label, and review the top of that list by hand. Estimate the error rate from the reviewed sample, fix what you can, and fix the labelling process that produced them.
Then they push further
Is it safe to just drop every example the model disagrees with?
No. The model may be wrong, and the hard, correct examples are exactly what it disagrees with. Dropping them teaches it only the easy cases. Review by hand, at least on a sample.
What gets marked up. Marked up for out-of-fold predictions, human review, and fixing the source of errors rather than only the data.
How would you check for train-test contamination?
For each test example, find its nearest training examples by exact hash, fuzzy text matching or embedding similarity, and read the closest pairs. Remove or reassign duplicates, then re-evaluate.
These are the questions for one topic. The whole question bank covers every topic on the site.