Eval-driven development, section 2 of 2, 4h

A week of improving one feature, worked through

By the end: You can run an improvement cycle end to end and recognise the decisions that make it work.

  • Starting from traces
  • Choosing the first fix by count
  • Keeping and reverting changes
  • Watching for overfitting
  • Writing it up

New to eval-driven development? A way of working rather than a tool. The eval suite runs automatically on every change, the way unit tests do, and the number it produces is what decides whether the change ships. Prompts, retrieval settings and model choices are then argued about with a score rather than with impressions. Why it matters

Everything in this topic has been described one piece at a time: traces, failure categories, evals, held-out splits, one change at a time. This section runs them together on a single feature across five days, with the numbers at each step, including the two changes that were measured and thrown away. It is here because the order of the work is the part that is hard to convey in the abstract, and the order is what makes it work.

The setup#

A support assistant answers questions about a software product from its help centre. The team says it 'feels unreliable'. Here is a realistic week, condensed.

Monday: read before measuring#

Read 120 recent traces and write a note on the first failure in each. Group the notes.

Answered from outdated article          17
Follow-up question lost its context     12
Invented a setting that does not exist   8
Too long, buried the answer              6
Other                                    5
                                        --
                                        48 failing of 120 (40%)

Tuesday: build the eval around those modes#

  • 80 cases from real traffic, including all 48 failures, split into 56 to tune on and 24 held out.
  • Code check for outdated sources: fail if any cited article is marked archived.
  • Pass or fail judge for invented settings, validated on 40 labels: catches 17 of 19 failures, accepts 20 of 21 passes.
  • Baseline: 22 of 56 on the tune set and 10 of 24 held out. That is 32 of 80, which matches the 48 known failures in the set.

Wednesday and Thursday: one change at a time#

Change                                         Tune set   Kept?
Exclude archived articles from the index         36/56     yes
Rewrite follow-ups into standalone questions     42/56     yes
Add 'only mention settings from the sources'     44/56     yes
Few-shot examples of short answers               45/56     no, +40% cost for +1 case
Switch to a larger model                         46/56     no, 3x cost

The first two changes were not prompt changes at all. The biggest wins in a system usually come from data and retrieval, which is why reading traces first matters.

Friday: check the held-out set once#

Held-out result: 17 of 24 pass, up from 10. The tune set went from 22 to 44 of 56 over the same changes. Both moved a lot, which suggests the changes generalise. If the tune set had jumped and the held-out set had barely moved, that would point to overfitting the tuning cases.

Is 10 to 17 of 24 real?

On 24 cases, check the disagreements. If seven cases went from fail to pass and none the other way, that is solid evidence even on a small set. If it was nine wins and two losses, the case is weaker, and growing the set is the next job.

Writing it up#

What we found: 40% of traces failed; top causes were outdated articles
  and lost follow-up context.
What we changed: excluded archived articles, added standalone rewriting,
  constrained settings to sources.
Result: held-out pass rate 10/24 -> 17/24. Cost per answer +8%, p95 +300 ms.
Not done: answer length (tried few-shot, not worth the cost).
Next: the eval set is small; add 100 cases from next week's traffic.

Where people go wrong

Making five changes in one go because each seemed obviously good. When the score moves, nobody knows why, and when it drops, nobody knows what to revert.

If this comes up in an interview

Tell me how you would improve an AI feature that 'feels unreliable'.

Read a hundred or so traces and categorise the failures, count them, and build an eval around the top modes with a held-out split. Then make one change at a time, starting with the most common failure, keep what improves the tune set at acceptable cost, and check the held-out set at the end.

Then they push further

Your first idea is to switch to a bigger model. Why might that be wrong?

If the failures come from outdated documents or lost context, a bigger model reads the same bad inputs. It also multiplies cost. Fix the causes the traces show first, then test a bigger model as one change among others.

How do you know you did not overfit to your eval?

Keep a held-out split you look at rarely, and check that it moves with the tuning set. Then watch production failure rates after release, since real traffic is the final held-out set.

What gets marked up. Strong answers start from data, change one thing at a time, weigh cost, and use a held-out set. This question often stands in for a behavioural round, so a real example with numbers scores highest.

Build this: Run this week on your own project: traces, failure table, eval with a held-out split, at least four single changes with a keep or revert decision each, and a write-up in the format above.

Watch it explained

43 min

Inspect - A LLM Eval Framework Used by Anthropic, DeepMind, Grok and More.

Hamel Husain

This is one section of Eval-driven development, 10 hours across 2 sections. Read it in full to get the whole topic in order.

Previous sectionImproving against a number

Stop 32 of 50 on the AI Engineer route

0 of 50 topics done · 0h behind you, 603h to go

Next on this routeTracing & observabilitySeeing what the system actually did.8h

What you have marked done is stored in this browser only. It does not reach a server, and clearing site data will clear it.