A code grader or an LLM judge

Good at

• Being deterministic: the same output always gets the same grade.
• Costing nothing to run, so it can go in CI on every commit.
• Checking what is genuinely checkable — valid JSON, schema conformance, the number, the tool call, the citation.
• Never drifting, so a score from March is comparable to a score today.
• Forcing you to define success precisely, which frequently improves the feature itself.

Costs you

• Only works where correctness is mechanical.
• Brittle to harmless variation — the right answer phrased differently fails.
• Assertions need maintaining as the output format legitimately evolves.
• Tempts you to evaluate what is easy to check instead of what matters.

Good at

• Qualities with no closed form: tone, helpfulness, faithfulness to a source, completeness.
• Scaling human judgement from fifty reviewed cases to five thousand graded ones.
• Pairwise comparison, where models agree with people far more than on absolute scores.
• Getting a signal on open-ended generation at all, where nothing else works.

Costs you

• It is a model, so it is wrong sometimes, and it is wrong in correlated ways.
• Known biases: position, length, self-preference, and generosity on fluent wrong answers.
• Costs tokens on every eval run, which is how eval suites become too expensive to run often.
• The judge itself needs validating against human labels, and revalidating when it is changed.
• Scores are not comparable across judge prompt or judge model versions.