Asked about Testing a system with a non-deterministic part

What is the difference between a test and an eval?

An answer that works

A test asserts something that must always hold and is deterministic. An eval measures a rate over a task set and is expected to be imperfect. Mixing them produces flaky tests that get deleted and evals nobody trusts.

Reading an answer is not being able to give one. The section that teaches this is Testing a system with a non-deterministic part, 6 hours of Testing.

Learn this properly

Also asked about Testing

How do you test code that calls an LLM?What is usually untested in an AI system and should not be?How do you test an agent without calling a model?What is property-based testing and where does it help in AI code?