Asked about Why evals exist
What is an eval?
An answer that works
A fixed set of inputs with a defined way of scoring the output, run on every change, producing a number you can compare between versions. It is the equivalent of a test suite for a system whose output is not deterministic.
Reading an answer is not being able to give one. The section that teaches this is Why evals exist, 3 hours of Evaluation.
Learn this properlyAlso asked about Evaluation
Why can you not just use unit tests?How would you start evaluating an existing feature with no evals?You join a team with an LLM feature and no evals. What do you do in the first week?Why note only the first failure in a trace?Why binary judges rather than a one to five scale?Where do eval cases come from?Why split an eval set into train and test?How big should an eval set be?
All 24 questions on Evaluation