Running judges on live traffic

A validated judge can be pointed at live traffic, which turns quality from something checked before release into something watched continuously. This section covers doing that honestly: sampling rather than judging everything, correcting the number for the judge's own known error rates, putting a range around it, and what to do when the judge model changes underneath you. The most valuable output is not the daily percentage but the list of traces it flagged, which is where you find out what is actually going wrong.

You do not need to judge everything#

A judge call on every request can cost as much as the feature itself. A random sample of a few hundred traces a day is enough to track a failure rate. Judge more where the stakes are higher, such as conversations that ended in an escalation.

A few hundred is a real number rather than a figure of speech, and it is worth deriving, because the sample size decides what changes you are able to notice at all. The uncertainty on a rate measured from a sample of n is the familiar square-root-of-p-times-one-minus-p-over-n, and what matters operationally is the second column: the smallest day-to-day move that is not just sampling.

PER_JUDGEMENT = 0.00375          # from the cost line in the LLM-as-judge topic
TRUE_FAIL = 0.05

print(f"{'sampled/day':>12}{'95% range on a 5% rate':>26}{'smallest change it can see':>28}{'cost/day':>10}")
for n in (100, 200, 500, 2000, 10000):
    se = (TRUE_FAIL * (1 - TRUE_FAIL) / n) ** 0.5
    detectable = 2 * 1.96 * se     # two independent days, both with this error
    print(f"{n:>12}{f'{TRUE_FAIL - 1.96 * se:.1%} to {TRUE_FAIL + 1.96 * se:.1%}':>26}"
          f"{detectable:>27.1%}{n * PER_JUDGEMENT:>10.2f}")

# sampled/day    95% range on a 5% rate  smallest change it can see  cost/day
#         100              0.7% to 9.3%                       8.5%      0.38
#         200              2.0% to 8.0%                       6.0%      0.75
#         500              3.1% to 6.9%                       3.8%      1.88
#        2000              4.0% to 6.0%                       1.9%      7.50
#       10000              4.6% to 5.4%                       0.9%     37.50

Five hundred a day is the answer for most systems, and the shape of the table is why: precision improves with the square root of the sample, so the first few hundred buy almost all of it. Two consequences follow. Do not bother alerting on a two-point move if you are sampling two hundred a day, because you cannot see one. And if you genuinely need to detect small moves, buy them with time rather than money, by comparing a seven-day window to the previous seven days instead of yesterday to the day before.

Sample by stratum rather than uniformly where the traffic is lopsided. A fixed share of everything, plus a much higher share of the categories that matter, gives you a usable number on the rare important cases without paying for the common ones. Weight the strata back together when you report the overall rate, or the number drifts every time the traffic mix changes.

The judge's own errors shift the number#

Say your judge catches 90% of real failures and accepts 95% of real passes, measured on your labels. If it marks 80% of today's traffic as passing, the true pass rate is not exactly 80%. You can correct for its known mistakes.

def corrected_pass_rate(judged_pass, accepts, catches):
    """
    judged_pass: share the judge marked PASS on unlabelled traffic
    accepts:     share of real passes the judge marks PASS
    catches:     share of real failures the judge marks FAIL
    """
    return (judged_pass + catches - 1) / (accepts + catches - 1)

print("raw judged pass rate      :", 0.80)
print("corrected                 :", round(corrected_pass_rate(0.80, 0.95, 0.90), 3))
for accepts, catches in ((0.95, 0.90), (0.95, 0.60), (0.85, 0.90)):
    print(f"  accepts {accepts:.0%}, catches {catches:.0%} -> "
          f"{corrected_pass_rate(0.80, accepts, catches):.3f}")

# raw judged pass rate      : 0.8
# corrected                 : 0.824
#   accepts 95%, catches 90% -> 0.824
#   accepts 95%, catches 60% -> 0.727
#   accepts 85%, catches 90% -> 0.933

Read the second and third rows as the warning they are. A judge that catches only 60% of failures makes the truth worse than the raw number said, and a judge that wrongly fails 15% of good outputs makes it much better. Getting the direction right is most of the value; getting the third decimal place is not available.

The correction only works when the judge is clearly better than chance, and it inherits any error in your measurements of accepts and catches. Those came from a limited number of labels, so the corrected number has more uncertainty than it looks.

Put an error bar on it#

A simple way to get one is the bootstrap: resample your labelled set and your daily sample with replacement many times, recompute the corrected rate each time, and take the middle 95% of the results as the range. Doing it properly means resampling both sources of error, the traffic sample and the labels behind the two rates, because either can dominate.

import numpy as np

def bootstrap(n_traffic=500, n_pass_labels=200, n_fail_labels=40,
              accepts=0.95, catches=0.90, judged_pass=0.80, boot=4000, seed=2):
    rng = np.random.default_rng(seed)
    out = []
    for _ in range(boot):
        jp = rng.binomial(n_traffic, judged_pass) / n_traffic
        a = rng.binomial(n_pass_labels, accepts) / n_pass_labels
        c = rng.binomial(n_fail_labels, catches) / n_fail_labels
        if a + c - 1 > 0.05:                 # the correction needs a usable judge
            out.append(corrected_pass_rate(jp, a, c))
    return np.percentile(out, [2.5, 97.5])

lo, hi = bootstrap()
print(f"95% range with 500 traffic samples and 40 labelled failures: "
      f"{lo:.3f} to {hi:.3f}")
lo, hi = bootstrap(n_fail_labels=200, n_pass_labels=1000)
print(f"95% range with 200 labelled failures and 1000 labelled passes : "
      f"{lo:.3f} to {hi:.3f}")
lo, hi = bootstrap(n_traffic=5000)
print(f"95% range with 5000 traffic samples and 40 labelled failures  : "
      f"{lo:.3f} to {hi:.3f}")

# 95% range with 500 traffic samples and 40 labelled failures: 0.768 to 0.877
# 95% range with 200 labelled failures and 1000 labelled passes : 0.779 to 0.866
# 95% range with 5000 traffic samples and 40 labelled failures  : 0.785 to 0.861

That is the result worth internalising, and it is not the one people expect. The corrected pass rate carries a band of roughly four points either side under generous sampling, and spending ten times more on either the labels or the daily sample barely moves it, because the two errors add. So report the corrected number as a trend with a band drawn on the chart, never as a target with a decimal place, and resist anyone who wants to set a threshold inside the band.

When the judge model changes#

If the provider updates the model behind your judge, or you switch judge models, its catch and accept rates change too. Pin the judge model version, and rerun it on your labelled set whenever it changes before comparing numbers across the change.

Use judges to find cases, not to punish them#

In production, the most valuable output of a judge is the list of traces it flagged. Read them. Some are real failures to add to the eval set, some reveal a weakness in the judge. The daily number is a trend line; the flagged traces are where you learn.

Go in knowing what that list contains, though, because the arithmetic of a rare event is unkind and it is the reason people give up on reading flags. A judge that catches 90% of failures and wrongly flags 5% of good outputs sounds accurate. What fraction of its flagged list is a real failure depends almost entirely on how often your system actually fails.

print(f"{'real failure rate':>18}{'flagged':>10}{'real ones in the list':>23}{'per 500 sampled':>18}")
for base in (0.20, 0.10, 0.05, 0.02, 0.01):
    catches, false_alarm = 0.90, 0.05
    flagged = base * catches + (1 - base) * false_alarm
    ppv = base * catches / flagged
    print(f"{base:>17.0%}{flagged:>10.1%}{ppv:>22.0%}"
          f"{round(flagged * 500):>13} to read")

#  real failure rate   flagged  real ones in the list   per 500 sampled
#               20%     22.0%                   82%          110 to read
#               10%     13.5%                   67%           68 to read
#                5%      9.2%                   49%           46 to read
#                2%      6.7%                   27%           34 to read
#                1%      5.9%                   15%           29 to read

Three things follow from that table. The first is not to describe the flagged list as failures to anyone, because on a good system most of it is not, and one round of a colleague reading twenty flags and finding seventeen acceptable answers is enough to end the practice. Call it the review queue and say what share of it is usually real.

The second is that the false-alarm rate matters more than the catch rate in production, which is the opposite of the priority during validation. Halving false alarms from 5% to 2.5% on a system that fails 2% of the time takes the list from 27% real to 42% real and shrinks it by a third. Raising the catch rate from 90% to 95% does almost nothing to either. If the queue is not being read, tighten the rubric against false alarms rather than adding sensitivity.

The third is that this is the argument for a second, stricter pass. Judge the sample cheaply, then re-judge only the flagged ones with a stronger model or a fuller rubric, and keep only what survives both. Two judges in series multiply their false-alarm rates while barely denting the catch rate, which is exactly the trade the table says you want, and the extra calls apply to 6% of the sample rather than all of it.

Noticing when the judge, rather than the system, moved#

Every number on this page assumes the instrument is fixed. It is not: the provider updates the model, someone improves the rubric, a colleague changes the temperature or the examples. Any of those shifts the catch and false-alarm rates, and the daily chart moves in a way indistinguishable from a quality change.

The defence is a fixed set of labelled examples kept next to the judge, thirty or so with known verdicts, run on every judging batch as a canary. It is a few pennies a day and it answers the question you will actually be asked when the line moves, which is whether anything changed on your side. If the canary verdicts are identical and the traffic number moved, the system moved. If the canary moved, nothing about the traffic number can be compared with yesterday until you have revalidated and said so on the chart.

• Pin the judge model version, the rubric text, the temperature and the examples, and treat a change to any of them as a change to the metric.
• Annotate the chart where the judge changed. A trend line with no annotations invites the wrong explanation for every step in it.
• Re-run the full labelled set, not just the canary, whenever the judge model changes, and publish the new catch and accept rates.
• Keep the old judge running in parallel for a week where you can afford it, which turns a discontinuity into an overlap you can measure.