Asked about Worked design: an agent acting on customer accounts
How would you evaluate this agent?
An answer that works
Against the end state of test accounts rather than the agent's words: whether the correct action happened, for the right amount, exactly once. Include adversarial cases where the customer tries to manipulate the agent into an unauthorised action.
Reading an answer is not being able to give one. The section that teaches this is Worked design: an agent acting on customer accounts, 6 hours of AI system design.
Learn this properlyAlso asked about AI system design
Design a system that answers questions over a company's internal documents.How do you handle the model being wrong in a production system?How would you know the system had regressed?How do you handle document permissions in a RAG system?What would you build first?How do you stop a support agent issuing unauthorised refunds?