Asked about Grading with a model
How do you know your LLM judge is any good?
An answer that works
Grade a sample by hand, making sure it contains real failures, and measure on labels the judge was not tuned on how often it catches the failures and how often it accepts the passes. Report those numbers alongside the score. Without them the score is unanchored. Poor results almost always mean the rubric is ambiguous rather than the judge being weak.
Then they push further
Your judge agrees with you 90% of the time. Good enough?
It depends where the disagreements are. If only 5% of outputs are failures, a judge that passes everything agrees 95% of the time and is useless. Measure separately how many real failures it catches and how many real passes it accepts.
How many labels do you need?
Enough that both the tuning half and the test half contain a reasonable number of failures, which usually means a few dozen of each. If failures are rare, go looking for them deliberately.
What gets marked up
Strong answers measure the judge against human labels on held-out data and look at failures and passes separately. Quoting overall agreement alone is marked down.
Reading an answer is not being able to give one. The section that teaches this is Grading with a model, 6 hours of LLM-as-judge.
Learn this properly