Asked about Grading with a model

What are the biases of an LLM judge?

An answer that works

Preference for length, preference for its own style, sensitivity to the order of compared options, and general leniency. Mitigate by ruling length out in the rubric, using a different judge model, running both orders, and giving concrete examples of failures.

Reading an answer is not being able to give one. The section that teaches this is Grading with a model, 6 hours of LLM-as-judge.

Learn this properly

Also asked about LLM-as-judge

How do you know your LLM judge is any good?When would you use pairwise comparison rather than scoring?Should the judge explain its reasoning?How would you monitor answer quality in production with an LLM judge?Your judge marks 80% of traffic as passing. Is the pass rate 80%?