Building evals first or shipping first
Do you build a measurement harness before the feature, or ship something and measure what comes back?
The short answer
Ship small with tracing on, read a hundred real traces, then build the eval set from the failures you found. Evals first is right when a wrong answer causes harm.
Evals first
Good at
- Making every later change decidable, instead of a matter of opinion.
- Catching regressions from a model upgrade before users find them.
- Forcing the team to agree what a good answer is, which is often the real disagreement.
- Enabling fast iteration, because a run takes minutes and nobody has to read outputs by hand.
- Protecting you at exactly the moment a deadline makes 'it looks fine' tempting.
Costs you
- Days spent before anything is in front of a user.
- Cases invented at a desk are not the cases users bring, so the first set measures the wrong thing.
- A suite built on the wrong assumptions is worse than none — it gives confident wrong numbers.
- Maintenance: an eval set that nobody prunes rots into noise.
Pick this when: You already understand the task well, or the cost of being wrong in production is high enough that discovering failures live is unacceptable.
Ship first, instrument heavily
Good at
- Finding out what users actually ask, which is reliably not what you guessed.
- Producing real failure cases, which are the only good raw material for an eval set.
- Delivering value while you learn, rather than after.
- Avoiding a suite that measures a task nobody turned out to want.
Costs you
- Users experience the failures, and some of those are not recoverable reputationally.
- Without tracing from day one you get complaints instead of data, and complaints are not debuggable.
- Without a baseline you cannot tell whether the next change helped.
- 'We will add evals later' has a well-documented completion rate.
- Rules out the whole class of features where a wrong answer causes real harm.
Pick this when: Low stakes, an internal or small audience, and tracing in place before launch rather than after.
Where the field disagrees
Practitioners split on this hard. One side, from the eval-driven-development tradition, holds that a team without a suite before launch never builds one and spends a year guessing. The other points out that most eval sets written before contact with users measure imagined problems, and that error analysis on real traces is a faster route to a good suite. They agree about tracing: nobody defends shipping without it.