Asked about Distillation in practice
Your feature uses a large model and costs too much. Would you distil it?
An answer that works
Perhaps, after trying cheaper options. First check whether a smaller model with a better prompt is good enough. If not, and the task is narrow and high volume, generate outputs from the large model on real inputs, filter them with evals, fine-tune a small model, and compare on a held-out set. Check the provider's terms first.
Then they push further
The distilled model matches the teacher on your eval but users report odd failures. Why?
The eval probably covers the common cases the training data covered, while the student is weaker on rare inputs the teacher handled by general ability. Add those failures to the eval and to the training data, or route unusual inputs to the large model.
What gets marked up
Marked up for comparing against a smaller prompted model first, filtering teacher outputs, and mentioning terms of use.
Reading an answer is not being able to give one. The section that teaches this is Distillation in practice, 4 hours of Compression & distillation.
Learn this properly