Asked about Shipping and measuring
How would you roll out a change to an AI feature?
An answer that works
Behind a flag, internal first, then one percent watching errors and latency, then ten and fifty percent for a real comparison, then full. Instant rollback at every stage, and the success metric written down before launch.
Then they push further
What metric decides whether you keep it?
One agreed before launch that reflects user benefit, such as tasks resolved without escalation, plus guardrails like complaint rate, cost and latency that must not get worse.
What gets marked up
Strong answers include offline evals, a staged rollout behind a flag, a metric agreed in advance with guardrails, and a fast way to roll back.
Reading an answer is not being able to give one. The section that teaches this is Shipping and measuring, 5 hours of Shipping and measuring.
Learn this properly