Asked about A structure for the design round
How would you know the system had regressed?
An answer that works
A fixed eval set run on every change, reporting quality, cost and latency together, with a held-out split. Plus production monitoring on the same signals, and sampling real traffic back into the eval set so it keeps describing what the system is actually asked.
Reading an answer is not being able to give one. The section that teaches this is A structure for the design round, 5 hours of AI system design.
Learn this properlyAlso asked about AI system design
Design a system that answers questions over a company's internal documents.How do you handle the model being wrong in a production system?How do you handle document permissions in a RAG system?What would you build first?How do you stop a support agent issuing unauthorised refunds?How would you evaluate this agent?