Asked about Evaluating agents and multi-step systems
What does it mean to find the first failing step, and why bother?
An answer that works
Walk each failed trace and mark the earliest step that went wrong. Tallied across runs, it shows which kind of step fails most, such as filling tool arguments. That points directly at the fix, where 'the agent is unreliable' does not.
Reading an answer is not being able to give one. The section that teaches this is Evaluating agents and multi-step systems, 5 hours of Evaluation.
Learn this properlyAlso asked about Evaluation
What is an eval?Why can you not just use unit tests?How would you start evaluating an existing feature with no evals?You join a team with an LLM feature and no evals. What do you do in the first week?Why note only the first failure in a trace?Why binary judges rather than a one to five scale?Where do eval cases come from?Why split an eval set into train and test?
All 24 questions on Evaluation