Model Distillation

Model Distillation, also known as Knowledge Distillation, is a Machine Learning technique in which a Student Model, usually smaller and more computationally efficient, is trained to approximate selected outputs, representations, or behavior of a Teacher Model. Popularized by Geoffrey Hinton and colleagues in 2015, building on earlier Model Compression research, the approach often involves minimizing a Loss Function that compares student predictions with teacher-provided Soft Targets. In classification, these targets are typically Probability Distributions produced through Softmax, and training may also incorporate ground-truth labels. Such targets convey Dark Knowledge: relationships among classes reflected in the teacher’s predictions beyond the highest-probability class. A student may retain much of the teacher’s performance, but preservation is not guaranteed and depends on the task, student capacity, training objective, and data. Distillation is an important Model Compression technique that can reduce Inference computation, memory requirements, and Latency for models such as Large Language Models and Convolutional Neural Networks. It can support deployment on resource-constrained Edge Computing devices, including those with limited GPU or VRAM resources, but feasibility also depends on model architecture, hardware, and complementary techniques such as Quantization and Pruning. NVIDIA provides distillation tooling and examples; Google DeepMind has published research using distillation; and Hugging Face publishes and supports distilled models such as DistilBERT and Distil-Whisper. These activities demonstrate support for the technique, rather than establishing broad production deployment by all three organizations.