Grading with a model

Some qualities cannot be checked in code. Whether an answer is faithful to its sources, whether the tone is right, whether a summary dropped the thing that mattered — no assertion covers those, and reading outputs by hand stops scaling after a few hundred. A judge is a model grading against a rubric you wrote. It is also a measuring instrument, and one you have never checked against your own labels tells you nothing you can act on. This section covers writing the rubric, the biases to design around, and producing the number that says how much to trust it.

Only for what code cannot check#

Exhaust deterministic checks first. Valid JSON, correct total, citation resolves, under the length limit, side effect happened. These are free, fast and cannot themselves be wrong. An LLM-as-judge is for what remains: faithfulness, helpfulness, tone.

That advice is easy to nod at and easy to ignore, because asking a judge feels more thorough than writing a function. The case worth seeing is one that sits exactly on the line: whether an answer cites a document that was actually retrieved. It sounds like a judgement about faithfulness and it is a set comparison. Below, the check defines the property, and a stub judge is asked the same question in words.

import numpy as np
rng = np.random.default_rng(12)

# 200 answers, each citing some document ids. Some cite an id that was never
# retrieved, which is a property a five-line function decides exactly.
N = 200
RETRIEVED = ["D-1", "D-2", "D-3", "D-4"]
answers = []
for i in range(N):
    cited = list(rng.choice(RETRIEVED, size=int(rng.integers(1, 4)), replace=False))
    if rng.random() < 0.15:
        cited.append("D-" + str(int(rng.integers(90, 99))))     # fabricated
    answers.append({"cited": cited})

def check(answer):
    """Exact, free, and it cannot have an opinion."""
    return set(answer["cited"]) <= set(RETRIEVED)

truth = np.array([check(a) for a in answers])          # here the check IS the truth

def stub_judge(truth, catch, false_alarm, rng):
    """A judge asked the same question in words. catch is how often it spots a
    fabricated id, false_alarm how often it objects to a clean answer."""
    r = rng.random(len(truth))
    return np.where(truth, r >= false_alarm, r >= catch)   # True means it passed

judged = stub_judge(truth, catch=0.82, false_alarm=0.05, rng=rng)

bad = ~truth
print("answers citing something not retrieved:", int(bad.sum()), "of", N)
print("the check   : found all", int(bad.sum()), "of them, by construction, for nothing")
print("the judge   :", int((judged == truth).sum()), "/", N, "correct")
print("             missed", int((judged & bad).sum()), "fabricated citations,",
      "objected to", int((~judged & truth).sum()), "clean answers")

IN_PER_M, OUT_PER_M = 3.00, 15.00
per_call = (950 / 1e6 * IN_PER_M) + (60 / 1e6 * OUT_PER_M)
print(f"judging costs ${per_call:.5f} per answer, ${per_call * N:.2f} for these {N},",
      f"${per_call * 50_000:.0f} for 50,000 a day")

# answers citing something not retrieved: 33 of 200
# the check   : found all 33 of them, by construction, for nothing
# the judge   : 179 / 200 correct
#              missed 9 fabricated citations, objected to 12 clean answers
# judging costs $0.00375 per answer, $0.75 for these 200, $188 for 50,000 a day

Note what the comparison is not. It is not that judges are unreliable in general; the catch and false-alarm rates used here are what a decent judge looks like. It is that on a checkable property those rates are pure loss, because the alternative has no error rate at all. The nine missed fabrications are the ones that matter: they are the failures you built the eval to find, and they are missed by a method you paid for.

The question to ask of any proposed judge, then, is what exactly it is being asked to decide, stated as a property rather than a quality. If the property can be computed from the output and the inputs, compute it. The grading section of the evals topic has a table of graders ordered by cost that is worth reading beside this: most of what people send to a judge is two rows above where they stopped.

The cost line is the other half of the decision and it is usually discovered late. A judge call carrying a 250-token rubric, a 400-token input and a 300-token answer, returning a short critique, is about 0.004 dollars. That is nothing per output and 188 dollars a day at fifty thousand outputs, which is why production judging is a sampling problem rather than a coverage problem, and why the topic's second section is about sampling.

A rubric is a specification#

Vague rubrics produce noise. The test is whether two careful people applying your rubric to the same output would reach the same verdict. If not, the model has no chance.

The most reliable form is a pass or fail question about one failure mode you have actually seen in your traces. A scale from one to five sounds more precise and is less useful: nobody agrees on the difference between a 3 and a 4, and an average of 3.6 tells you nothing about what to fix.

Weak:   "Rate the quality of this summary from 1 to 5."

Usable:
  You are checking one thing: does the summary state anything
  the source does not support? Ignore style, length and tone.

  FAIL if any claim is missing from the source or contradicts it.
  PASS if every claim can be found in the source, even if worded
  differently.

  Example of a FAIL: source says revenue rose 4%, summary says
  "revenue rose sharply".

  Write one or two sentences of critique, then the verdict on its
  own line: PASS or FAIL.

• One failure mode per judge. A single verdict on correctness, tone and length together cannot be interpreted.
• Say explicitly what to ignore, or the judge will quietly weigh it.
• Include a real example of a pass and a fail from your own data, especially a borderline one.
• Ask for a short critique before the verdict. It usually improves accuracy, and when the judge is wrong the critique shows you why. Keep it short, since it costs tokens on every call.

Measure it against your own labels#

# human and judge are lists of True (pass) / False (fail) on the held-out half.
# Thirty-six outputs, twelve of them failures you labelled yourself.
human = [True] * 24 + [False] * 12
judge = [True] * 22 + [False] * 2 + [True] * 3 + [False] * 9

pairs = list(zip(human, judge))
real_fails  = [j for h, j in pairs if not h]
real_passes = [j for h, j in pairs if h]

catches = sum(not j for j in real_fails) / len(real_fails)
accepts = sum(j for j in real_passes) / len(real_passes)
print(f"catches {catches:.0%} of real failures, accepts {accepts:.0%} of real passes")

# catches 75% of real failures, accepts 92% of real passes

Once you know those two rates you can also correct the judge's pass rate on unlabelled data for its known mistakes, instead of taking its raw number at face value.

When agreement is poor, the rubric is almost always the problem rather than the judge. Look at the disagreements: they usually reveal that the rubric was ambiguous about a case you had not considered.

Biases to design around#

• Length. Judges prefer longer answers. Say length is not a factor, and check by scoring a deliberately verbose answer.
• Self-preference. A model favours output in its own style, so consider a different model as judge.
• Position. In a comparison, the order affects the verdict. Run both orders and average.
• Leniency. Judges pass too much. Include a concrete fail example, and watch the catch rate on real failures.

Comparison is easier than scoring#

Asking which of two answers is better is more reliable than asking for an absolute score, because the judge does not need a calibrated scale. It is the better tool when comparing two versions of a system, which is usually what you are doing.

How much better is worth knowing, because it decides how many labels a decision costs you. The obstacle in any comparison is that tasks differ from each other far more than two versions of a system differ, so most of what you measure is which tasks you happened to draw. Three designs handle that differently: scoring each system on its own sample of tasks, scoring both on the same tasks, and showing both answers to the judge together.

import numpy as np

# Two systems on the same tasks. B is genuinely a little better. The hard part
# is that tasks differ in difficulty far more than the systems differ from each
# other, which is true of every eval set anyone has built.
DELTA, TASK_SPREAD = 0.05, 1.0
NOISE_ABS, NOISE_REL = 0.40, 0.25   # a judge is steadier comparing than scoring

def picks_the_better_system(n, rng, method):
    task = rng.normal(0, TASK_SPREAD, n)
    if method == "pointwise, different items":
        a = task + rng.normal(0, NOISE_ABS, n)
        b = rng.normal(0, TASK_SPREAD, n) + DELTA + rng.normal(0, NOISE_ABS, n)
        return b.mean() > a.mean()
    if method == "pointwise, same items":
        a = task + rng.normal(0, NOISE_ABS, n)
        b = task + DELTA + rng.normal(0, NOISE_ABS, n)
        return b.mean() > a.mean()
    # pairwise: one comparative judgement per item, so the task term cancels
    return (DELTA + rng.normal(0, NOISE_REL, n)).mean() > 0

rng = np.random.default_rng(21)
METHODS = ("pointwise, different items", "pointwise, same items", "pairwise")
print(f"{'items':>6}" + "".join(f"{m:>28}" for m in METHODS))
for n in (25, 50, 100, 200, 400, 800):
    cells = "".join(f"{np.mean([picks_the_better_system(n, rng, m) for _ in range(4000)]):>27.0%} "
                    for m in METHODS)
    print(f"{n:>6}{cells}")
print("(share of runs that pick the genuinely better system)")

# items  pointwise, different items       pointwise, same items                    pairwise
#     25                        56%                         68%                         84%
#     50                        59%                         73%                         92%
#    100                        63%                         82%                         98%
#    200                        69%                         89%                        100%
#    400                        75%                         96%                        100%
#    800                        84%                         99%                        100%

Two separate effects are visible in that table and it is worth keeping them apart. Going from different items to the same items is free and it is most of the win: you were measuring the task sample, and running both versions over the same tasks removes it. Going from scoring separately to comparing together is the rest, and it depends on the assumption in the second constant, that a judge is steadier comparing two answers than placing one on a scale. That is a property of your judge, not a law, and the way to find your own number is to judge the same pairs twice and see how often each method contradicts itself.

So the choice is not about which is more accurate in the abstract. Use pairwise when the question is which of two versions to ship, because it answers that question with a quarter of the labels. Use pointwise when you need a number that means something on its own: a rate you can track week to week, alert on, or put in front of someone who was not in the room. A pairwise result cannot do that, since it says only that this beat that, against an opponent that will not exist next month.

• Pairwise: choosing between two prompts, two models, two retrieval settings. Cheapest route to a decision.
• Pointwise pass or fail: a failure rate you can track over time, alert on, and compare across releases.
• Pointwise with a score out of five: rarely. Nobody agrees what a 3 is, and the average tells you nothing about what to fix.
• Neither: the property is computable. Write the function.

One practical warning about pairwise. Which answer is shown first changes the verdict, by enough to reverse it, so every comparison has to be run in both orders and averaged. That doubles the cost and it is not optional. The grading section of the evals topic measures the size of that effect and shows why the flip rate is the wrong way to quantify it.

Which model does the judging#

A judge prefers writing that looks like its own. This is the bias people most often mention and least often measure, partly because measuring it needs a design rather than a probe: one judge cannot tell you about its own preferences. What works is a crossed comparison. Take two systems built on two different model families, get them to genuine parity, and have judges from both families and from a third grade the same pairs.

import numpy as np

# Two systems written on two model families, genuinely equal in quality, and
# three judges. The crossed design is what makes self-preference visible: each
# judge grades both, so the interaction is the bias.
N = 2000
PULL = 0.12          # how far a judge leans towards output in its own style

def compare(judge_family, n, rng):
    """Probability that the answer from family X wins, as this judge sees it."""
    p = 0.50 + (PULL if judge_family == "X" else -PULL if judge_family == "Y" else 0.0)
    return rng.random(n) < p

rng = np.random.default_rng(31)
jx = compare("X", N, rng)
jy = compare("Y", N, rng)
jz = compare("Z", N, rng)

print("system X written on family X, system Y on family Y, truly equal")
print(f"judge from family X : X wins {jx.mean():.1%}")
print(f"judge from family Y : X wins {jy.mean():.1%}")
print(f"judge from family Z : X wins {jz.mean():.1%}")
print(f"average of X and Y  : X wins {(jx.mean() + jy.mean()) / 2:.1%}")
print(f"self-preference gap : {jx.mean() - jy.mean():.1%} "
      f"(twice the pull, because both judges lean)")

# system X written on family X, system Y on family Y, truly equal
# judge from family X : X wins 59.9%
# judge from family Y : X wins 38.6%
# judge from family Z : X wins 48.6%
# average of X and Y  : X wins 49.3%
# self-preference gap : 21.2% (twice the pull, because both judges lean)

A twelve-point lean produces a twenty-one point swing between the two verdicts, because both judges are leaning and they lean in opposite directions. If you are choosing between a prompt on your current model and one on a candidate model, and the judge is your current model, that swing is larger than any difference you are likely to be measuring. It is also invisible from inside: the judge is consistent, confident, and gives good reasons.

• Comparing systems on the same model family: self-preference mostly cancels, and you can use that family as judge.
• Comparing across families, or evaluating a migration: use a third family, or average two judges from the families being compared.
• Either way, pin the judge model and version. It is part of the instrument, and an upgrade invalidates the comparison with last month.
• Cheaper judge models are usually fine for a pass or fail question on one failure mode, and noticeably worse at open comparison. Measure yours against your labels before deciding, since the saving is real and so is the drop.

Everything above is about choosing and configuring the instrument. Proving it works is a separate exercise, and it is the one that makes the number mean anything: label a sample yourself, measure agreement above chance rather than raw agreement, and size the labelled set by counting failures rather than labels. The grading section of the evals topic works through that protocol with the arithmetic, and it is the natural next read after this one.