LLM-as-judge

Using a model to grade outputs against a rubric, where no code check can decide correctness. Cheap and scalable, and itself something that has to be validated against human labels.

What people get wrong

Judges have systematic biases: for longer answers, for their own family's style, and for whichever option came first. Measure agreement with human graders before trusting the score.

A definition is not an understanding. This is taught properly in Grading with a model, part of LLM-as-judge.

Learn this properly