Asked about Choosing and changing models
How would you choose between two models?
An answer that works
Use benchmarks only to shortlist, then run my own eval on real tasks and compare pass rate, cost and p95 latency together. Keep the prompt identical for the first comparison, then tune per model, since the ranking can change once each gets a fair prompt.
Then they push further
The cheaper model scores 3 points lower on your eval. Which do you pick?
Check first that 3 points is not noise. Then look at which cases it fails and what those failures cost. If they are low stakes and the saving is large, route those requests to the cheaper model. If they include costly errors, the saving is not worth it.
What gets marked up
Marked up for deciding on your own eval, cost and latency rather than public benchmarks, and for looking at which cases differ, not just the totals.
Reading an answer is not being able to give one. The section that teaches this is Choosing and changing models, 6 hours of Choosing a model.
Learn this properly