Distillation

Also: knowledge distillation

Training a small model to reproduce the outputs of a large one, rather than training it on the original labels. The teacher's full probability distribution carries more signal than a single correct answer.