Compression & distillation interview questions
5 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Making a model smaller
Read the section that teaches thisIf this comes up in an interview
What is knowledge distillation?
Training a small student model to reproduce a large teacher's outputs. Learning from the teacher's full probability distribution carries more information than hard labels, so the student can approach the teacher's quality at a fraction of the size.
Why does unstructured pruning often not speed anything up?
It zeroes individual weights but leaves the tensor shape unchanged, so the hardware still processes the zeros. Structured pruning removes whole units or layers, which is a genuine speedup at a larger quality cost.
How do you decide whether a compressed model is acceptable?
Measure on your own eval rather than trusting published aggregates, because degradation is uneven across capabilities. Then compare size, latency and quality together, which makes it a product decision with numbers rather than an argument.
Distillation in practice
Read the section that teaches thisIf this comes up in an interview
Your feature uses a large model and costs too much. Would you distil it?
Perhaps, after trying cheaper options. First check whether a smaller model with a better prompt is good enough. If not, and the task is narrow and high volume, generate outputs from the large model on real inputs, filter them with evals, fine-tune a small model, and compare on a held-out set. Check the provider's terms first.
Then they push further
The distilled model matches the teacher on your eval but users report odd failures. Why?
The eval probably covers the common cases the training data covered, while the student is weaker on rare inputs the teacher handled by general ability. Add those failures to the eval and to the training data, or route unusual inputs to the large model.
What gets marked up. Marked up for comparing against a smaller prompted model first, filtering teacher outputs, and mentioning terms of use.
Why use temperature in distillation?
At normal temperature the teacher's distribution is very peaked, so it carries little information beyond the top label. Higher temperature spreads it out so the student can learn the relative probabilities of the other classes.
These are the questions for one topic. The whole question bank covers every topic on the site.