Asked about Why evals exist
Why can you not just use unit tests?
An answer that works
Model output varies between runs and there is usually no single correct string. Exact-match assertions flake and get deleted. Evals score properties and use graded judgement rather than equality.
Reading an answer is not being able to give one. The section that teaches this is Why evals exist, 3 hours of Evaluation.
Learn this properlyAlso asked about Evaluation
What is an eval?How would you start evaluating an existing feature with no evals?You join a team with an LLM feature and no evals. What do you do in the first week?Why note only the first failure in a trace?Why binary judges rather than a one to five scale?Where do eval cases come from?Why split an eval set into train and test?How big should an eval set be?
All 24 questions on Evaluation