Asked about Changing models without breaking things
Your provider is retiring the model you use in three months. What is your plan?
An answer that works
Pick candidates, run the eval set on each, adapt the prompt for the best one, shadow real traffic, then roll out gradually with the old model still available. Start early, because prompt changes and review take longer than expected.
Then they push further
The new model scores the same on your eval but support tickets rise after launch. What happened?
The eval did not cover what changed. Common gaps are tone, length, refusals, or rare input types. Read the tickets and the traces behind them, add those cases to the eval, and roll back if the problem is serious.
What gets marked up
Marked up for eval comparison, prompt adaptation, shadow traffic and a staged rollout with rollback. Swapping the model name and shipping is marked down.
Reading an answer is not being able to give one. The section that teaches this is Changing models without breaking things, 4 hours of Choosing a model.
Learn this properly