Asked about Running judges on live traffic

Your judge marks 80% of traffic as passing. Is the pass rate 80%?

An answer that works

Not exactly. The judge misses some failures and wrongly fails some passes. With its catch and accept rates measured on labelled data you can correct the estimate, and you should report the uncertainty that comes with those measurements.

Reading an answer is not being able to give one. The section that teaches this is Running judges on live traffic, 4 hours of LLM-as-judge.

Learn this properly

Also asked about LLM-as-judge

How do you know your LLM judge is any good?What are the biases of an LLM judge?When would you use pairwise comparison rather than scoring?Should the judge explain its reasoning?How would you monitor answer quality in production with an LLM judge?