How do you use production traces to improve an AI system?

An answer that works

Keep failures, slow and expensive runs, and a sample of normal ones. Categorise failures and fix the largest category first, and add each real failure to the eval set so it cannot silently return.