Asked about Error analysis: read traces before you write evals
You join a team with an LLM feature and no evals. What do you do in the first week?
An answer that works
Get full traces logged if they are not already. Read about a hundred real ones, write a note on the first failure in each, group the notes into failure modes and count them. Fix the obvious bugs directly, then build a small eval per recurring failure mode, code checks where possible and a validated judge where not. Only then start changing prompts.
Then they push further
Why not start with standard metrics like faithfulness and relevance?
Because they are chosen before you know how this system fails. They produce numbers that look meaningful and often do not move when the real problem gets better or worse. Failure modes found in your own traces are specific enough to fix and to test.
How do you know a hundred traces is enough?
You do not pick the number in advance with confidence. You keep reading until new traces stop revealing new failure modes. A hundred is a starting point that usually gets most of the common ones.
What if the product team disagrees with your labels?
Then the disagreement is the finding. It means 'good' was never defined. Get one domain owner to decide, write the decision into the failure mode definitions, and label against those.
What gets marked up
Strong answers start from reading data, not from choosing a tool or a metric, and they mention counting failures to prioritise. Weak answers name an eval framework in the first sentence.
Reading an answer is not being able to give one. The section that teaches this is Error analysis: read traces before you write evals, 6 hours of Evaluation.
Learn this properlyAlso asked about Evaluation