Asked about Improving against a number

How do you improve an AI system methodically?

An answer that works

Score a baseline, make one change, score again, keep or revert, repeat. Hold out a test split you never inspect during iteration and report that number, and track cost alongside quality since most improvements are trades.

Reading an answer is not being able to give one. The section that teaches this is Improving against a number, 6 hours of Eval-driven development.

Learn this properly

Also asked about Eval-driven development

Why hold out a test split when you are only tuning prompts?How do you know when to stop improving?Tell me how you would improve an AI feature that 'feels unreliable'.