Asked about Error analysis: read traces before you write evals
Why binary judges rather than a one to five scale?
An answer that works
A binary question about one failure mode can be checked against human labels and has a clear meaning. A five point scale has no agreed meaning between adjacent points, drifts between runs and between graders, and nobody can act on an average of 3.4.
What gets marked up
Marked up if the candidate ties this to validation: you can measure how often a binary judge agrees with your labels on both passes and failures.
Reading an answer is not being able to give one. The section that teaches this is Error analysis: read traces before you write evals, 6 hours of Evaluation.
Learn this properlyAlso asked about Evaluation
What is an eval?Why can you not just use unit tests?How would you start evaluating an existing feature with no evals?You join a team with an LLM feature and no evals. What do you do in the first week?Why note only the first failure in a trace?Where do eval cases come from?Why split an eval set into train and test?How big should an eval set be?
All 24 questions on Evaluation