LLM-as-judge

Some qualities cannot be checked in code. Whether an answer is faithful to its sources, whether the tone is right, whether a summary dropped the thing that mattered — no assertion covers those, and reading outputs by hand stops scaling after a few hundred. A judge is a model grading against a rubric you wrote. It is also a measuring instrument, and one you have never checked against your own labels tells you nothing you can act on. This section covers writing the rubric, the biases to design around, and producing the number that says how much to trust it.

Only for what code cannot check#

Exhaust deterministic checks first. Valid JSON, correct total, citation resolves, under the length limit, side effect happened. These are free, fast and cannot themselves be wrong. An LLM-as-judge is for what remains: faithfulness, helpfulness, tone.

That advice is easy to nod at and easy to ignore, because asking a judge feels more thorough than writing a function. The case worth seeing is one that sits exactly on the line: whether an answer cites a document that was actually retrieved. It sounds like a judgement about faithfulness and it is a set comparison. Below, the check defines the property, and a stub judge is asked the same question in words.

import numpy as np
rng = np.random.default_rng(12)

# 200 answers, each citing some document ids. Some cite an id that was never
# retrieved, which is a property a five-line function decides exactly.
N = 200
RETRIEVED = ["D-1", "D-2", "D-3", "D-4"]
answers = []
for i in range(N):
    cited = list(rng.choice(RETRIEVED, size=int(rng.integers(1, 4)), replace=False))
    if rng.random() < 0.15:
        cited.append("D-" + str(int(rng.integers(90, 99))))     # fabricated
    answers.append({"cited": cited})

def check(answer):
    """Exact, free, and it cannot have an opinion."""
    return set(answer["cited"]) <= set(RETRIEVED)

truth = np.array([check(a) for a in answers])          # here the check IS the truth

def stub_judge(truth, catch, false_alarm, rng):
    """A judge asked the same question in words. catch is how often it spots a
    fabricated id, false_alarm how often it objects to a clean answer."""
    r = rng.random(len(truth))
    return np.where(truth, r >= false_alarm, r >= catch)   # True means it passed

judged = stub_judge(truth, catch=0.82, false_alarm=0.05, rng=rng)

bad = ~truth
print("answers citing something not retrieved:", int(bad.sum()), "of", N)
print("the check   : found all", int(bad.sum()), "of them, by construction, for nothing")
print("the judge   :", int((judged == truth).sum()), "/", N, "correct")
print("             missed", int((judged & bad).sum()), "fabricated citations,",
      "objected to", int((~judged & truth).sum()), "clean answers")

IN_PER_M, OUT_PER_M = 3.00, 15.00
per_call = (950 / 1e6 * IN_PER_M) + (60 / 1e6 * OUT_PER_M)
print(f"judging costs ${per_call:.5f} per answer, ${per_call * N:.2f} for these {N},",
      f"${per_call * 50_000:.0f} for 50,000 a day")

# answers citing something not retrieved: 33 of 200
# the check   : found all 33 of them, by construction, for nothing
# the judge   : 179 / 200 correct
#              missed 9 fabricated citations, objected to 12 clean answers
# judging costs $0.00375 per answer, $0.75 for these 200, $188 for 50,000 a day

Note what the comparison is not. It is not that judges are unreliable in general; the catch and false-alarm rates used here are what a decent judge looks like. It is that on a checkable property those rates are pure loss, because the alternative has no error rate at all. The nine missed fabrications are the ones that matter: they are the failures you built the eval to find, and they are missed by a method you paid for.

The question to ask of any proposed judge, then, is what exactly it is being asked to decide, stated as a property rather than a quality. If the property can be computed from the output and the inputs, compute it. The grading section of the evals topic has a table of graders ordered by cost that is worth reading beside this: most of what people send to a judge is two rows above where they stopped.

The cost line is the other half of the decision and it is usually discovered late. A judge call carrying a 250-token rubric, a 400-token input and a 300-token answer, returning a short critique, is about 0.004 dollars. That is nothing per output and 188 dollars a day at fifty thousand outputs, which is why production judging is a sampling problem rather than a coverage problem, and why the topic's second section is about sampling.

A rubric is a specification#

Vague rubrics produce noise. The test is whether two careful people applying your rubric to the same output would reach the same verdict. If not, the model has no chance.

The most reliable form is a pass or fail question about one failure mode you have actually seen in your traces. A scale from one to five sounds more precise and is less useful: nobody agrees on the difference between a 3 and a 4, and an average of 3.6 tells you nothing about what to fix.

Weak:   "Rate the quality of this summary from 1 to 5."

Usable:
  You are checking one thing: does the summary state anything
  the source does not support? Ignore style, length and tone.

  FAIL if any claim is missing from the source or contradicts it.
  PASS if every claim can be found in the source, even if worded
  differently.

  Example of a FAIL: source says revenue rose 4%, summary says
  "revenue rose sharply".

  Write one or two sentences of critique, then the verdict on its
  own line: PASS or FAIL.

• One failure mode per judge. A single verdict on correctness, tone and length together cannot be interpreted.
• Say explicitly what to ignore, or the judge will quietly weigh it.
• Include a real example of a pass and a fail from your own data, especially a borderline one.
• Ask for a short critique before the verdict. It usually improves accuracy, and when the judge is wrong the critique shows you why. Keep it short, since it costs tokens on every call.

Measure it against your own labels#

# human and judge are lists of True (pass) / False (fail) on the held-out half.
# Thirty-six outputs, twelve of them failures you labelled yourself.
human = [True] * 24 + [False] * 12
judge = [True] * 22 + [False] * 2 + [True] * 3 + [False] * 9

pairs = list(zip(human, judge))
real_fails  = [j for h, j in pairs if not h]
real_passes = [j for h, j in pairs if h]

catches = sum(not j for j in real_fails) / len(real_fails)
accepts = sum(j for j in real_passes) / len(real_passes)
print(f"catches {catches:.0%} of real failures, accepts {accepts:.0%} of real passes")

# catches 75% of real failures, accepts 92% of real passes

Once you know those two rates you can also correct the judge's pass rate on unlabelled data for its known mistakes, instead of taking its raw number at face value.

When agreement is poor, the rubric is almost always the problem rather than the judge. Look at the disagreements: they usually reveal that the rubric was ambiguous about a case you had not considered.

Biases to design around#

• Length. Judges prefer longer answers. Say length is not a factor, and check by scoring a deliberately verbose answer.
• Self-preference. A model favours output in its own style, so consider a different model as judge.
• Position. In a comparison, the order affects the verdict. Run both orders and average.
• Leniency. Judges pass too much. Include a concrete fail example, and watch the catch rate on real failures.

Comparison is easier than scoring#

Asking which of two answers is better is more reliable than asking for an absolute score, because the judge does not need a calibrated scale. It is the better tool when comparing two versions of a system, which is usually what you are doing.

How much better is worth knowing, because it decides how many labels a decision costs you. The obstacle in any comparison is that tasks differ from each other far more than two versions of a system differ, so most of what you measure is which tasks you happened to draw. Three designs handle that differently: scoring each system on its own sample of tasks, scoring both on the same tasks, and showing both answers to the judge together.

import numpy as np

# Two systems on the same tasks. B is genuinely a little better. The hard part
# is that tasks differ in difficulty far more than the systems differ from each
# other, which is true of every eval set anyone has built.
DELTA, TASK_SPREAD = 0.05, 1.0
NOISE_ABS, NOISE_REL = 0.40, 0.25   # a judge is steadier comparing than scoring

def picks_the_better_system(n, rng, method):
    task = rng.normal(0, TASK_SPREAD, n)
    if method == "pointwise, different items":
        a = task + rng.normal(0, NOISE_ABS, n)
        b = rng.normal(0, TASK_SPREAD, n) + DELTA + rng.normal(0, NOISE_ABS, n)
        return b.mean() > a.mean()
    if method == "pointwise, same items":
        a = task + rng.normal(0, NOISE_ABS, n)
        b = task + DELTA + rng.normal(0, NOISE_ABS, n)
        return b.mean() > a.mean()
    # pairwise: one comparative judgement per item, so the task term cancels
    return (DELTA + rng.normal(0, NOISE_REL, n)).mean() > 0

rng = np.random.default_rng(21)
METHODS = ("pointwise, different items", "pointwise, same items", "pairwise")
print(f"{'items':>6}" + "".join(f"{m:>28}" for m in METHODS))
for n in (25, 50, 100, 200, 400, 800):
    cells = "".join(f"{np.mean([picks_the_better_system(n, rng, m) for _ in range(4000)]):>27.0%} "
                    for m in METHODS)
    print(f"{n:>6}{cells}")
print("(share of runs that pick the genuinely better system)")

# items  pointwise, different items       pointwise, same items                    pairwise
#     25                        56%                         68%                         84%
#     50                        59%                         73%                         92%
#    100                        63%                         82%                         98%
#    200                        69%                         89%                        100%
#    400                        75%                         96%                        100%
#    800                        84%                         99%                        100%

Two separate effects are visible in that table and it is worth keeping them apart. Going from different items to the same items is free and it is most of the win: you were measuring the task sample, and running both versions over the same tasks removes it. Going from scoring separately to comparing together is the rest, and it depends on the assumption in the second constant, that a judge is steadier comparing two answers than placing one on a scale. That is a property of your judge, not a law, and the way to find your own number is to judge the same pairs twice and see how often each method contradicts itself.

So the choice is not about which is more accurate in the abstract. Use pairwise when the question is which of two versions to ship, because it answers that question with a quarter of the labels. Use pointwise when you need a number that means something on its own: a rate you can track week to week, alert on, or put in front of someone who was not in the room. A pairwise result cannot do that, since it says only that this beat that, against an opponent that will not exist next month.

• Pairwise: choosing between two prompts, two models, two retrieval settings. Cheapest route to a decision.
• Pointwise pass or fail: a failure rate you can track over time, alert on, and compare across releases.
• Pointwise with a score out of five: rarely. Nobody agrees what a 3 is, and the average tells you nothing about what to fix.
• Neither: the property is computable. Write the function.

One practical warning about pairwise. Which answer is shown first changes the verdict, by enough to reverse it, so every comparison has to be run in both orders and averaged. That doubles the cost and it is not optional. The grading section of the evals topic measures the size of that effect and shows why the flip rate is the wrong way to quantify it.

Which model does the judging#

A judge prefers writing that looks like its own. This is the bias people most often mention and least often measure, partly because measuring it needs a design rather than a probe: one judge cannot tell you about its own preferences. What works is a crossed comparison. Take two systems built on two different model families, get them to genuine parity, and have judges from both families and from a third grade the same pairs.

import numpy as np

# Two systems written on two model families, genuinely equal in quality, and
# three judges. The crossed design is what makes self-preference visible: each
# judge grades both, so the interaction is the bias.
N = 2000
PULL = 0.12          # how far a judge leans towards output in its own style

def compare(judge_family, n, rng):
    """Probability that the answer from family X wins, as this judge sees it."""
    p = 0.50 + (PULL if judge_family == "X" else -PULL if judge_family == "Y" else 0.0)
    return rng.random(n) < p

rng = np.random.default_rng(31)
jx = compare("X", N, rng)
jy = compare("Y", N, rng)
jz = compare("Z", N, rng)

print("system X written on family X, system Y on family Y, truly equal")
print(f"judge from family X : X wins {jx.mean():.1%}")
print(f"judge from family Y : X wins {jy.mean():.1%}")
print(f"judge from family Z : X wins {jz.mean():.1%}")
print(f"average of X and Y  : X wins {(jx.mean() + jy.mean()) / 2:.1%}")
print(f"self-preference gap : {jx.mean() - jy.mean():.1%} "
      f"(twice the pull, because both judges lean)")

# system X written on family X, system Y on family Y, truly equal
# judge from family X : X wins 59.9%
# judge from family Y : X wins 38.6%
# judge from family Z : X wins 48.6%
# average of X and Y  : X wins 49.3%
# self-preference gap : 21.2% (twice the pull, because both judges lean)

A twelve-point lean produces a twenty-one point swing between the two verdicts, because both judges are leaning and they lean in opposite directions. If you are choosing between a prompt on your current model and one on a candidate model, and the judge is your current model, that swing is larger than any difference you are likely to be measuring. It is also invisible from inside: the judge is consistent, confident, and gives good reasons.

• Comparing systems on the same model family: self-preference mostly cancels, and you can use that family as judge.
• Comparing across families, or evaluating a migration: use a third family, or average two judges from the families being compared.
• Either way, pin the judge model and version. It is part of the instrument, and an upgrade invalidates the comparison with last month.
• Cheaper judge models are usually fine for a pass or fail question on one failure mode, and noticeably worse at open comparison. Measure yours against your labels before deciding, since the saving is real and so is the drop.

Everything above is about choosing and configuring the instrument. Proving it works is a separate exercise, and it is the one that makes the number mean anything: label a sample yourself, measure agreement above chance rather than raw agreement, and size the labelled set by counting failures rather than labels. The grading section of the evals topic works through that protocol with the arithmetic, and it is the natural next read after this one.

A validated judge can be pointed at live traffic, which turns quality from something checked before release into something watched continuously. This section covers doing that honestly: sampling rather than judging everything, correcting the number for the judge's own known error rates, putting a range around it, and what to do when the judge model changes underneath you. The most valuable output is not the daily percentage but the list of traces it flagged, which is where you find out what is actually going wrong.

You do not need to judge everything#

A judge call on every request can cost as much as the feature itself. A random sample of a few hundred traces a day is enough to track a failure rate. Judge more where the stakes are higher, such as conversations that ended in an escalation.

A few hundred is a real number rather than a figure of speech, and it is worth deriving, because the sample size decides what changes you are able to notice at all. The uncertainty on a rate measured from a sample of n is the familiar square-root-of-p-times-one-minus-p-over-n, and what matters operationally is the second column: the smallest day-to-day move that is not just sampling.

PER_JUDGEMENT = 0.00375          # from the cost line in the LLM-as-judge topic
TRUE_FAIL = 0.05

print(f"{'sampled/day':>12}{'95% range on a 5% rate':>26}{'smallest change it can see':>28}{'cost/day':>10}")
for n in (100, 200, 500, 2000, 10000):
    se = (TRUE_FAIL * (1 - TRUE_FAIL) / n) ** 0.5
    detectable = 2 * 1.96 * se     # two independent days, both with this error
    print(f"{n:>12}{f'{TRUE_FAIL - 1.96 * se:.1%} to {TRUE_FAIL + 1.96 * se:.1%}':>26}"
          f"{detectable:>27.1%}{n * PER_JUDGEMENT:>10.2f}")

# sampled/day    95% range on a 5% rate  smallest change it can see  cost/day
#         100              0.7% to 9.3%                       8.5%      0.38
#         200              2.0% to 8.0%                       6.0%      0.75
#         500              3.1% to 6.9%                       3.8%      1.88
#        2000              4.0% to 6.0%                       1.9%      7.50
#       10000              4.6% to 5.4%                       0.9%     37.50

Five hundred a day is the answer for most systems, and the shape of the table is why: precision improves with the square root of the sample, so the first few hundred buy almost all of it. Two consequences follow. Do not bother alerting on a two-point move if you are sampling two hundred a day, because you cannot see one. And if you genuinely need to detect small moves, buy them with time rather than money, by comparing a seven-day window to the previous seven days instead of yesterday to the day before.

Sample by stratum rather than uniformly where the traffic is lopsided. A fixed share of everything, plus a much higher share of the categories that matter, gives you a usable number on the rare important cases without paying for the common ones. Weight the strata back together when you report the overall rate, or the number drifts every time the traffic mix changes.

The judge's own errors shift the number#

Say your judge catches 90% of real failures and accepts 95% of real passes, measured on your labels. If it marks 80% of today's traffic as passing, the true pass rate is not exactly 80%. You can correct for its known mistakes.

def corrected_pass_rate(judged_pass, accepts, catches):
    """
    judged_pass: share the judge marked PASS on unlabelled traffic
    accepts:     share of real passes the judge marks PASS
    catches:     share of real failures the judge marks FAIL
    """
    return (judged_pass + catches - 1) / (accepts + catches - 1)

print("raw judged pass rate      :", 0.80)
print("corrected                 :", round(corrected_pass_rate(0.80, 0.95, 0.90), 3))
for accepts, catches in ((0.95, 0.90), (0.95, 0.60), (0.85, 0.90)):
    print(f"  accepts {accepts:.0%}, catches {catches:.0%} -> "
          f"{corrected_pass_rate(0.80, accepts, catches):.3f}")

# raw judged pass rate      : 0.8
# corrected                 : 0.824
#   accepts 95%, catches 90% -> 0.824
#   accepts 95%, catches 60% -> 0.727
#   accepts 85%, catches 90% -> 0.933

Read the second and third rows as the warning they are. A judge that catches only 60% of failures makes the truth worse than the raw number said, and a judge that wrongly fails 15% of good outputs makes it much better. Getting the direction right is most of the value; getting the third decimal place is not available.

The correction only works when the judge is clearly better than chance, and it inherits any error in your measurements of accepts and catches. Those came from a limited number of labels, so the corrected number has more uncertainty than it looks.

Put an error bar on it#

A simple way to get one is the bootstrap: resample your labelled set and your daily sample with replacement many times, recompute the corrected rate each time, and take the middle 95% of the results as the range. Doing it properly means resampling both sources of error, the traffic sample and the labels behind the two rates, because either can dominate.

import numpy as np

def bootstrap(n_traffic=500, n_pass_labels=200, n_fail_labels=40,
              accepts=0.95, catches=0.90, judged_pass=0.80, boot=4000, seed=2):
    rng = np.random.default_rng(seed)
    out = []
    for _ in range(boot):
        jp = rng.binomial(n_traffic, judged_pass) / n_traffic
        a = rng.binomial(n_pass_labels, accepts) / n_pass_labels
        c = rng.binomial(n_fail_labels, catches) / n_fail_labels
        if a + c - 1 > 0.05:                 # the correction needs a usable judge
            out.append(corrected_pass_rate(jp, a, c))
    return np.percentile(out, [2.5, 97.5])

lo, hi = bootstrap()
print(f"95% range with 500 traffic samples and 40 labelled failures: "
      f"{lo:.3f} to {hi:.3f}")
lo, hi = bootstrap(n_fail_labels=200, n_pass_labels=1000)
print(f"95% range with 200 labelled failures and 1000 labelled passes : "
      f"{lo:.3f} to {hi:.3f}")
lo, hi = bootstrap(n_traffic=5000)
print(f"95% range with 5000 traffic samples and 40 labelled failures  : "
      f"{lo:.3f} to {hi:.3f}")

# 95% range with 500 traffic samples and 40 labelled failures: 0.768 to 0.877
# 95% range with 200 labelled failures and 1000 labelled passes : 0.779 to 0.866
# 95% range with 5000 traffic samples and 40 labelled failures  : 0.785 to 0.861

That is the result worth internalising, and it is not the one people expect. The corrected pass rate carries a band of roughly four points either side under generous sampling, and spending ten times more on either the labels or the daily sample barely moves it, because the two errors add. So report the corrected number as a trend with a band drawn on the chart, never as a target with a decimal place, and resist anyone who wants to set a threshold inside the band.

When the judge model changes#

If the provider updates the model behind your judge, or you switch judge models, its catch and accept rates change too. Pin the judge model version, and rerun it on your labelled set whenever it changes before comparing numbers across the change.

Use judges to find cases, not to punish them#

In production, the most valuable output of a judge is the list of traces it flagged. Read them. Some are real failures to add to the eval set, some reveal a weakness in the judge. The daily number is a trend line; the flagged traces are where you learn.

Go in knowing what that list contains, though, because the arithmetic of a rare event is unkind and it is the reason people give up on reading flags. A judge that catches 90% of failures and wrongly flags 5% of good outputs sounds accurate. What fraction of its flagged list is a real failure depends almost entirely on how often your system actually fails.

print(f"{'real failure rate':>18}{'flagged':>10}{'real ones in the list':>23}{'per 500 sampled':>18}")
for base in (0.20, 0.10, 0.05, 0.02, 0.01):
    catches, false_alarm = 0.90, 0.05
    flagged = base * catches + (1 - base) * false_alarm
    ppv = base * catches / flagged
    print(f"{base:>17.0%}{flagged:>10.1%}{ppv:>22.0%}"
          f"{round(flagged * 500):>13} to read")

#  real failure rate   flagged  real ones in the list   per 500 sampled
#               20%     22.0%                   82%          110 to read
#               10%     13.5%                   67%           68 to read
#                5%      9.2%                   49%           46 to read
#                2%      6.7%                   27%           34 to read
#                1%      5.9%                   15%           29 to read

Three things follow from that table. The first is not to describe the flagged list as failures to anyone, because on a good system most of it is not, and one round of a colleague reading twenty flags and finding seventeen acceptable answers is enough to end the practice. Call it the review queue and say what share of it is usually real.

The second is that the false-alarm rate matters more than the catch rate in production, which is the opposite of the priority during validation. Halving false alarms from 5% to 2.5% on a system that fails 2% of the time takes the list from 27% real to 42% real and shrinks it by a third. Raising the catch rate from 90% to 95% does almost nothing to either. If the queue is not being read, tighten the rubric against false alarms rather than adding sensitivity.

The third is that this is the argument for a second, stricter pass. Judge the sample cheaply, then re-judge only the flagged ones with a stronger model or a fuller rubric, and keep only what survives both. Two judges in series multiply their false-alarm rates while barely denting the catch rate, which is exactly the trade the table says you want, and the extra calls apply to 6% of the sample rather than all of it.

Noticing when the judge, rather than the system, moved#

Every number on this page assumes the instrument is fixed. It is not: the provider updates the model, someone improves the rubric, a colleague changes the temperature or the examples. Any of those shifts the catch and false-alarm rates, and the daily chart moves in a way indistinguishable from a quality change.

The defence is a fixed set of labelled examples kept next to the judge, thirty or so with known verdicts, run on every judging batch as a canary. It is a few pennies a day and it answers the question you will actually be asked when the line moves, which is whether anything changed on your side. If the canary verdicts are identical and the traffic number moved, the system moved. If the canary moved, nothing about the traffic number can be compared with yesterday until you have revalidated and said so on the chart.

• Pin the judge model version, the rubric text, the temperature and the examples, and treat a change to any of them as a change to the metric.
• Annotate the chart where the judge changed. A trend line with no annotations invites the wrong explanation for every step in it.
• Re-run the full labelled set, not just the canary, whenever the judge model changes, and publish the new catch and accept rates.
• Keep the old judge running in parallel for a week where you can afford it, which turns a discontinuity into an overlap you can measure.