Asked about Making a model smaller

What is knowledge distillation?

An answer that works

Training a small student model to reproduce a large teacher's outputs. Learning from the teacher's full probability distribution carries more information than hard labels, so the student can approach the teacher's quality at a fraction of the size.

Reading an answer is not being able to give one. The section that teaches this is Making a model smaller, 6 hours of Compression & distillation.

Learn this properly

Also asked about Compression & distillation

Why does unstructured pruning often not speed anything up?How do you decide whether a compressed model is acceptable?Your feature uses a large model and costs too much. Would you distil it?Why use temperature in distillation?