Error analysis: read traces before you write evals

This is the first step of evaluation and almost everyone skips it, which is why so many eval suites produce tidy numbers about nothing in particular. The work is manual: sit with a sample of real runs, describe each failure in your own words, then group and count those descriptions until a ranked list of failure modes falls out. Everything you build afterwards is shaped by that list, so this section decides whether the rest of your evaluation is aimed at anything.

The mistake this section prevents#

Most people start evaluation by picking metrics: helpfulness, faithfulness, tone. They wire up a library that scores all three, get numbers that look reasonable, and learn nothing. The metrics were chosen before anyone looked at how the system actually fails, so they measure things that were never the problem.

The order that works is the reverse. Read what the system did, write down what went wrong in plain words, group those notes, count them, and only then decide what to measure. The evals you end up with are shaped like your real failures instead of like a generic checklist.

What a trace is#

A trace is the full record of one request: the user input, the system prompt, every model call with its exact input and output, every tool call with its arguments and result, and the final answer. If you only log the final answer you cannot tell whether a wrong reply came from bad retrieval, a bad tool result, or the model ignoring good context.

trace = {
    "id": "t_0192",
    "input": "Can I get a refund on the annual plan after 20 days?",
    "steps": [
        {"type": "retrieve", "query": "annual plan refund", "hits": ["policy#4", "faq#12"]},
        {"type": "llm", "prompt_version": "v7", "output": "Yes, within 30 days..."},
    ],
    "final": "Yes, you can get a full refund within 30 days.",
    "latency_ms": 2140,
    "cost_usd": 0.0031,
}

Three fields in there earn their place for reasons that are not obvious until you need them. The prompt version, because half of what you find will turn out to be one prompt revision that shipped on a Tuesday. The retrieval hits by id, because that is what separates the model ignoring the right document from the system never fetching it. And the cost, because a failure mode that is cheap and a failure mode that burns two dollars a run are different priorities even when they happen equally often.

What you cannot recover later is worth listing, since most teams discover it the week they start this work. Raw model output before any parsing or post-processing, because the bug is often in your parser. Tool arguments exactly as sent, not as rendered in a log line. The system prompt as it actually went out, variables filled in. Anything the user did afterwards: retried, rephrased, escalated, left. And a stable id per conversation, so you can find the trace when a customer quotes one sentence of it back to you.

Which hundred traces to read#

Not the first hundred, and not the hundred with the worst feedback. The first hundred are one hour of one day and carry whatever was happening that hour. The worst-rated hundred tell you about the failures users bother to report, which is a biased and small subset — most people just leave.

Take a stratified sample, the way the dataset section builds an eval set, and then add the obvious suspects on top as a separate pile. Keep the two piles apart in your notes, because they answer different questions: the stratified pile tells you how often things go wrong, and the suspect pile tells you how badly.

• A random draw within each intent or category, with a floor so rare categories are present at all.
• Every conversation the user escalated, retried three times, or abandoned mid-way.
• Every trace where a code check already fails: unparseable output, a citation that resolves to nothing, a tool error.
• The longest and the most expensive runs, which are where loops and thrashing hide.
• A handful of ordinary successful traces, so you calibrate on what normal looks like and do not start seeing failures everywhere.

Step one: open coding#

Pull a sample of traces, around a hundred to start. For each one, read it end to end and write one short note about the first thing that went wrong. Not a score. A sentence, in your own words, specific to that trace.

t_0192  Policy says 14 days for annual plans. Retrieved the monthly plan section instead.
t_0193  Fine.
t_0194  Answered in English, user wrote in German.
t_0195  Promised a refund the agent has no power to issue.
t_0196  Fine.
t_0197  Retrieved the right section, then quoted the wrong number from it.

Note only the first failure in each trace. Later errors are often caused by the first one, and counting them separately inflates problems that fix themselves.

Step two: group notes into failure modes#

Once you have notes, read them together and cluster the ones that describe the same underlying problem. Give each cluster a name that someone could act on. 'Bad answer' is not a failure mode. 'Retrieved the section for the wrong plan type' is.

Failure mode                              Count   Share
Retrieved the wrong plan's policy            14     31%
Promised actions the bot cannot take          9     20%
Quoted wrong number from correct source       7     16%
Wrong language                                6     13%
Refused a question it could answer            5     11%
Other                                         4      9%
                                             --
                                             45   failing of 100 traces

Two rules make that table stable rather than a matter of mood. A mode has to name what went wrong, not how bad it was, so that a fix is implied by the name. And a mode only exists once three traces belong to it; below that it stays in an 'other' pile you re-read at the end, when you usually find that four of them were the same thing described four ways.

Then weight the counts by what each one costs#

Counting is the right first move and the wrong last one. The most frequent mode is often a minor annoyance and the rarest is often the one that ends up in a regulator's inbox. Put a cost per occurrence beside the count, multiply by the traffic the sample was drawn from, and the ordering usually changes.

# Counted from 100 read traces, with what one occurrence costs the business.
MODES = [
    ("retrieved the wrong plan's policy", 14, 4.0),
    ("promised an action the bot cannot take", 9, 30.0),
    ("quoted the wrong number from the right source", 7, 12.0),
    ("replied in the wrong language", 6, 2.0),
    ("refused a question it could answer", 5, 3.0),
    ("leaked another customer's order id", 1, 900.0),
]
TRAFFIC = 40_000        # conversations a month

print(f"{'failure mode':<46}{'count':>6}{'cost':>8}{'per month':>12}")
rows = []
for name, count, unit in MODES:
    monthly = count / 100 * TRAFFIC * unit
    rows.append((monthly, name, count, unit))
for monthly, name, count, unit in sorted(rows, reverse=True):
    print(f"{name:<46}{count:>6}{unit:>8.0f}{monthly:>12,.0f}")

# failure mode                                   count    cost   per month
# leaked another customer's order id                 1     900     360,000
# promised an action the bot cannot take             9      30     108,000
# quoted the wrong number from the right source      7      12      33,600
# retrieved the wrong plan's policy                 14       4      22,400
# refused a question it could answer                 5       3       6,000
# replied in the wrong language                      6       2       4,800

Do not over-engineer the cost column. Support minutes, refunds issued, churn risk and regulatory exposure in whatever unit you can defend, rounded hard, agreed with whoever owns the product. Its job is to stop the loudest count from winning automatically, and a rough figure does that job.

Step three: decide what each mode needs#

Not every failure deserves an eval. Some are just bugs. If the wrong language comes from a missing instruction, add the instruction and move on. Build evals for failures that are likely to come back when someone edits the prompt, changes the model, or touches retrieval.

• Fix directly: a missing instruction, a broken tool, a template bug. Then add one regression case.
• Code check: anything a function can verify, such as the language of the reply or whether a number matches the source.
• Judge: anything that needs reading, such as whether the bot promised something it cannot do. One judge per failure mode, answering pass or fail.

If you have no users yet#

You still need traces. Generate realistic inputs deliberately rather than asking a model for 'fifty test questions', which gives you fifty variations of the same polite easy question. Pick the dimensions along which real inputs differ, list values for each, and combine them.

import itertools, random

personas  = ["new customer", "long-time customer", "angry customer", "non-native speaker"]
topics    = ["refund", "plan change", "invoice error", "account access"]
scenarios = ["clear question", "missing key detail", "asks for something not allowed", "two questions at once"]

combos = random.sample(list(itertools.product(personas, topics, scenarios)), 60)
# Turn each combo into a natural message: by hand for the first ten,
# then with a model, then read every one and delete the unrealistic ones.

Run the system on those inputs, save full traces, and do the same open coding. Replace synthetic traces with real ones as soon as you have any.

When to stop reading#

Keep reading until new traces stop producing new failure modes. When twenty traces in a row only add to categories you already have, you have seen most of what this sample can show you. Repeat the whole exercise after any large change, and on a small sample of production traffic every week.

That rule of thumb has a number behind it, and the number is easy. The share of your traces showing a failure mode you have seen exactly once estimates the chance that the next trace shows a mode you have never seen at all. It is called the Good-Turing estimate, it was invented for a code-breaking problem with the same shape, and it takes one line. Here is a simulated system with eighteen real failure modes, a few common and a long tail, so the estimate can be checked against a truth you would not normally have.

import numpy as np
rng = np.random.default_rng(5)

# A made-up system with 18 real failure modes, a few common and a long tail.
weights = np.array([1 / (i + 1) ** 1.1 for i in range(18)])
weights = weights / weights.sum()

def read_traces(n):
    """Which failure mode each of n failing traces turned out to show."""
    return rng.choice(len(weights), size=n, p=weights)

print(f"{'traces read':>12}{'modes found':>13}{'seen once':>11}{'new-mode rate':>15}{'truth':>8}")
for n in (20, 40, 80, 160, 320):
    sample = read_traces(n)
    counts = np.bincount(sample, minlength=len(weights))
    found = int((counts > 0).sum())
    singletons = int((counts == 1).sum())
    # Good-Turing: the share of traces showing a mode exactly once estimates
    # the chance the next trace shows a mode you have never seen.
    estimate = singletons / n
    truth = weights[counts == 0].sum()
    print(f"{n:>12}{found:>13}{singletons:>11}{estimate:>15.1%}{truth:>8.1%}")

#  traces read  modes found  seen once  new-mode rate   truth
#           20           10          6          30.0%   24.4%
#           40           12          4          10.0%   16.0%
#           80           16          4           5.0%    3.3%
#          160           18          3           1.9%    0.0%
#          320           18          0           0.0%    0.0%

Two honest limits. The estimate only counts modes you can distinguish, so it goes to zero the moment you stop looking carefully, and a lazy reader reaches saturation quickly and wrongly. And it says nothing about modes that cannot appear in this sample at all, such as the failures of a feature that launches next month. Treat it as a stopping rule for this round of reading, not a certificate that the system has eighteen problems.

One person who understands the domain should own the labels. A committee produces disagreements about what 'good' means and no labels. That person can be you.

That said, have a second person label thirty of the same traces once, early. You are about to build graders that encode these definitions, and if two careful people disagree about which mode a trace belongs to, no rubric written from those definitions can be consistent either. Measure that disagreement the same way the grading section measures a judge, with a kappa rather than a raw agreement percentage, and fix the definitions before writing a single grader. Half an hour spent here saves the week that otherwise goes into tuning a judge against labels that were never stable.