Asked about Grading: code first, judges second
Your judge scores 4.2 out of 5. How much do you trust that?
An answer that works
Not much, for two reasons. Without a measured comparison against human grading the number is unanchored. And an average on a five point scale does not say what is wrong. I would replace it with pass or fail judges for specific failure modes, validated against my own labels, so the result reads as 'unsupported claims in 12% of summaries' rather than 4.2.
Reading an answer is not being able to give one. The section that teaches this is Grading: code first, judges second, 6 hours of Evaluation.
Learn this properlyAlso asked about Evaluation
What is an eval?Why can you not just use unit tests?How would you start evaluating an existing feature with no evals?You join a team with an LLM feature and no evals. What do you do in the first week?Why note only the first failure in a trace?Why binary judges rather than a one to five scale?Where do eval cases come from?Why split an eval set into train and test?
All 24 questions on Evaluation