Using traces to improve the system

Tracing gives you a record of what the system did. This section is about what comes after that: deciding which traces are worth keeping, grouping failures so you are looking at a count rather than an anecdote, and turning each real failure into an eval case so it cannot come back unnoticed. Done regularly, this is what stops an eval set from being something you invented once and turns it into a description of what the system is actually asked to do.

What to keep#

Keeping every trace in full is expensive and usually unnecessary. Keep all failures and slow or costly runs, plus a random sample of normal ones, and everything for a period after any change.

Sampling has one rule that is easy to get wrong: the decision must be made once per run, at the top, and inherited by every span inside it. Sampling each span independently gives you traces with holes in them, which are worse than no trace at all because they look complete. Hash the run id and keep the run if the hash falls below the rate, which also means the same run is sampled or not consistently across services.

One table, and the questions it answers#

Once traces are structured, a day of them is a table with one row per run. Everything in this section is an aggregation of that table, so it is worth having one to work on. The rows below are synthetic and shaped like real traffic: three features with different turn counts, usage concentrated in a few heavy users, and a latency distribution with a tail.

import numpy as np
rng = np.random.default_rng(0)
N = 20000

# One row per run, the shape a day of traces actually has.
feature = rng.choice(["support-chat", "doc-summary", "code-review"], N, p=[.6, .3, .1])
w = 1.0 / (np.arange(1, 2001) ** 1.1)          # real usage is not uniform
user = rng.choice(2000, N, p=w / w.sum())
turns = np.where(feature == "code-review", rng.poisson(6, N) + 1, rng.poisson(1, N) + 1)
in_tok = turns * rng.integers(2000, 9000, N)
out_tok = turns * rng.integers(100, 700, N)
cost = (in_tok * 5.0 + out_tok * 25.0) / 1e6           # usd, large model rates
latency = turns * rng.lognormal(6.6, 0.55, N)          # ms

print("latency per run, milliseconds")
print(f"  mean {latency.mean():7.0f}   p50 {np.percentile(latency, 50):7.0f}"
      f"   p95 {np.percentile(latency, 95):7.0f}   p99 {np.percentile(latency, 99):7.0f}"
      f"   max {latency.max():7.0f}")
# latency per run, milliseconds
#   mean    2116   p50    1425   p95    6120   p99   11424   max   32633

Report percentiles, not averages, and pick the percentile from who is complaining. The p50 is what a typical user sees. The p95 is what an unlucky user sees several times a week and is usually where the complaints come from. The p99 is what an agentic run with five model calls hits routinely, because a run is only as fast as its slowest step and five draws from that tail is a different distribution from one. That last point is why agent latency surprises people: the tail multiplies.

Cost attribution#

A monthly bill is one number and tells you nothing you can act on. The same money split by feature and by user tells you where to spend engineering time, and the two splits usually point at different things.

print("\ncost attribution")
print("feature          runs   usd    share   usd per run")
for f in ("support-chat", "doc-summary", "code-review"):
    m = feature == f
    print(f"{f:15s}{m.sum():6d}{cost[m].sum():7.2f}{100*cost[m].sum()/cost.sum():7.1f}%"
          f"{cost[m].mean():12.4f}")

per_user = np.bincount(user, weights=cost)
order = np.sort(per_user)[::-1]
top1 = int(len(order) * 0.01)
print(f"\ntop 1% of users ({top1}) account for {100*order[:top1].sum()/cost.sum():.1f}% of spend")
print(f"the median user costs {np.median(per_user):.3f} usd, the heaviest {order[0]:.2f} usd")

cost attribution
feature          runs   usd    share   usd per run
support-chat    11944 892.06   47.9%      0.0747
doc-summary      6096 466.29   25.0%      0.0765
code-review      1960 504.07   27.1%      0.2572

top 1% of users (20) account for 54.1% of spend
the median user costs 0.147 usd, the heaviest 315.24 usd

Both halves of that change a decision. Per-run cost identifies the feature worth optimising, and it is not the one with the most traffic: shaving 20% off support-chat saves less than halving the number of turns code review takes. Per-user cost identifies whether you have a pricing problem rather than an efficiency problem, and a single user at 315 usd against a median of 0.15 is the kind of thing that is invisible in an aggregate and obvious in a trace. Recording user and feature on the root span costs two strings and is what makes either query possible; adding them afterwards means the last three months cannot be analysed.

What to alert on, and what only to record#

The distinction is whether a human should be woken. Record everything; alert only on something a person can act on now. The usual mistake is alerting on individual failures, which in any system with a normal background error rate produces a stream nobody reads, and the real incident then arrives as one more line in a channel everyone has muted.

# Background failure rate of 1%, rising to 6% at run 15000 when something breaks.
rate = np.where(np.arange(N) < 15000, 0.01, 0.06)
failed = rng.random(N) < rate
window = 500
roll = np.convolve(failed.astype(float), np.ones(window) / window, mode="valid")
over = np.flatnonzero(roll > 0.03) + window
print(f"\n{failed.sum()} failures in {N} runs")
print(f"alerting on each failure:              {failed.sum()} notifications")
print(f"alerting on the rate over {window} runs:    "
      f"crosses 3% at run {over[0]}, {over[0] - 15000} runs after the break")
print(f"windows above 3% before the break:     {(over < 15000).sum()}")
# 466 failures in 20000 runs
# alerting on each failure:              466 notifications
# alerting on the rate over 500 runs:    crosses 3% at run 15181, 181 runs after the break
# windows above 3% before the break:     0

The window is the only parameter and it trades one thing against one thing: a shorter window notices sooner and fires on noise, a longer window is quiet and slow. The 500-run window above waits 181 runs, and a 100-run window would have caught it sooner at the cost of firing on quiet periods where three failures happened to land together. Pick it from your traffic rate, so that the window covers a few minutes at your busiest and you are not alerting on a sample of four overnight.

• Alert on: error rate over a window, p95 latency above the number you promised, spend per hour above a ceiling, and a queue that is growing rather than draining.
• Alert on: cached tokens falling to zero across a deployment, which is a large and otherwise silent cost increase.
• Record only: individual failures, individual slow runs, token counts, per-user cost, tool result sizes. These are what you read when an alert fires, not reasons to fire one.
• Never alert on something with no action attached. An alert whose runbook is 'have a look' trains people to ignore alerts.

Look for patterns, not anecdotes#

One bad trace is a story. A hundred traces grouped by failure type is a priority list. Categorise failures, count them, and fix the largest category first.

kinds = ["tool_timeout", "empty_retrieval", "max_tokens", "schema_invalid", "refusal"]
idx = rng.choice(5, failed.sum(), p=[.42, .27, .16, .10, .05])
counts = np.bincount(idx, minlength=5)
print("\nfailures grouped, largest first")
running = 0
for k in np.argsort(-counts):
    running += counts[k]
    print(f"  {kinds[k]:16s}{counts[k]:5d}{100*counts[k]/counts.sum():7.1f}%"
          f"   cumulative {100*running/counts.sum():5.1f}%")
# failures grouped, largest first
#   tool_timeout      212   45.5%   cumulative  45.5%
#   empty_retrieval   104   22.3%   cumulative  67.8%
#   max_tokens         80   17.2%   cumulative  85.0%
#   refusal            36    7.7%   cumulative  92.7%
#   schema_invalid     34    7.3%   cumulative 100.0%

The categories come from reading traces, and the first pass is done by hand: read fifty failures, write down what went wrong in a few words each, and the categories will name themselves. Automate the classification afterwards, once the labels exist, rather than inventing labels in advance. A category with one member is a note, not a category.

Traces become eval cases#

User feedback#

Thumbs up and down, corrections, and whether a user retried or abandoned are cheap signals attached to real traces. They are noisy and biased towards people who bother to click, and they point you at where to look.