Asked about Running an AI service
What would you monitor on an AI service?
An answer that works
Latency at p50 and p95 split by step, error rate broken down by type, cost per request and per hour with an alert, token usage including cached tokens to confirm caching works, queue depth, and upstream provider health.
Reading an answer is not being able to give one. The section that teaches this is Running an AI service, 5 hours of Deployment & serving.
Learn this properlyAlso asked about Deployment & serving
What is different about deploying an AI service?Your model provider is having an outage. What should your service do?Your prediction endpoint has a 150 ms p99 target and is missing it. Where do you start?How should prompt changes be released?An AI feature starts misbehaving in production. What do you do first?How long should a canary run before you trust it?What breaks a model in production that no offline evaluation can catch?You expect 50 requests per second at peak, each taking about 8 seconds. What capacity do you need?
All 10 questions on Deployment & serving