Asked about Running an AI service
Your model provider is having an outage. What should your service do?
An answer that works
Retry briefly with backoff, then fall back to an alternative model, then serve cached results if reasonable, then return an honest message. Decide this before the outage, because during one you will not.
Reading an answer is not being able to give one. The section that teaches this is Running an AI service, 5 hours of Deployment & serving.
Learn this properlyAlso asked about Deployment & serving
What is different about deploying an AI service?What would you monitor on an AI service?Your prediction endpoint has a 150 ms p99 target and is missing it. Where do you start?How should prompt changes be released?An AI feature starts misbehaving in production. What do you do first?How long should a canary run before you trust it?What breaks a model in production that no offline evaluation can catch?You expect 50 requests per second at peak, each taking about 8 seconds. What capacity do you need?
All 10 questions on Deployment & serving