Asked about Knowing whether it works
How many eval tasks do you need?
An answer that works
Twenty real ones beats two hundred synthetic ones. Draw them from actual usage, include every failure you have already seen in production, and version the set so a score from last month still means something.
Reading an answer is not being able to give one. The section that teaches this is Knowing whether it works, 4 hours of The agent loop.
Learn this properlyAlso asked about The agent loop
What is an agent, in one sentence?When would you not use an agent?Where does the loop run?How do you stop an agent running forever?An agent keeps calling the same tool with the same arguments. What do you do?What should an agent return when it runs out of budget?An agent overflows its context after ten turns. What do you do?What must survive compaction?
All 12 questions on The agent loop