Asked about Grading: code first, judges second
You are comparing two prompt versions with a pairwise judge. What do you do about position bias?
An answer that works
Run every comparison twice with the answers in both orders, and average. The gap between the win rate in one order and the other is the size of the bias, and it is worth reporting as a property of the judge. In a realistic case a fifteen-point pull towards whichever answer is shown first is enough to reverse the verdict on a version that is genuinely better, so judging in one fixed order can send you the wrong way.
Reading an answer is not being able to give one. The section that teaches this is Grading: code first, judges second, 6 hours of Evaluation.
Learn this properlyAlso asked about Evaluation
What is an eval?Why can you not just use unit tests?How would you start evaluating an existing feature with no evals?You join a team with an LLM feature and no evals. What do you do in the first week?Why note only the first failure in a trace?Why binary judges rather than a one to five scale?Where do eval cases come from?Why split an eval set into train and test?
All 24 questions on Evaluation