Asked about Where ground truth comes from
Two annotators disagree on 30 percent of labels. What does that tell you?
An answer that works
That the task definition is ambiguous, not that someone is careless. It also caps achievable model performance at around that agreement rate, since there is no stable target above it. The fix is clearer guidelines, starting from the disagreements.
Then they push further
Another team reports 95 per cent agreement. Is their task better defined?
Not necessarily, and on a rare-class task almost certainly not. Two annotators who ignore the item entirely and guess at the prevalence agree 95 per cent of the time when the positive class is 2 per cent, because they both say no to nearly everything. Ask for Cohen's kappa and the class prevalence; kappa subtracts the agreement expected from the marginals, so chance agreement scores zero.
You rewrote the guidelines and kappa did not move. What now?
That usually means the disagreement was genuine ambiguity rather than unclear wording, so the label set is wrong rather than the prose. If two careful readers keep splitting on the same kind of item, that kind of item is a third category, and adding it, or adding an explicit not-applicable, recovers more agreement than further clarification.
What gets marked up
Marked up for treating disagreement as a definition problem, for knowing raw agreement is inflated by class imbalance, and for naming kappa.
Reading an answer is not being able to give one. The section that teaches this is Where ground truth comes from, 5 hours of Data quality & labelling.
Learn this properlyAlso asked about Data quality & labelling