Shipping AI Mar 2015

Distilling the Knowledge in a Neural Network

Geoffrey Hinton et al. · NeurIPS 2014 Deep Learning Workshop

arXiv:1503.02531

In short

A small model learns from a big one by matching its full, softened output probabilities, not just the right answer. Those “soft targets” carry what the big model knows about which wrong answers are nearly right, so the small model gets much of its accuracy at a fraction of the cost.

Why it matters

Distillation is how big models become fast, cheap ones you can actually deploy.

Read first

The 4 Field Guide ideas this paper leans on.

Neural Networks · Landmark Softmax A function that turns a list of scores (logits) into probabilities that are all positive and sum to 1; the standard output of classifiers and language models. Training · Reference Teacher Model The larger, more accurate model in knowledge distillation that guides student training. Training · Reference Student Model The smaller model in knowledge distillation learning to mimic the teacher's behavior. Training · Standard Knowledge Distillation Training a smaller 'student' model to mimic a larger 'teacher' model, transferring knowledge while reducing size.

Starting from scratch? The full route 9 ideas · basics first

Machine Learning ✓ understood
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
Neural Network ✓ understood
A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.
Activation Function ✓ understood
A non-linear function applied to neuron outputs that introduces non-linearity, enabling networks to learn complex patterns.
Softmax · read first ✓ understood
A function that turns a list of scores (logits) into probabilities that are all positive and sum to 1; the standard output of classifiers and language models.
Dataset ✓ understood
A collection of data examples used for training, validating, or testing machine learning models.
Training ✓ understood
The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.
Knowledge Distillation · read first ✓ understood
Training a smaller 'student' model to mimic a larger 'teacher' model, transferring knowledge while reducing size.
Teacher Model · read first ✓ understood
The larger, more accurate model in knowledge distillation that guides student training.
Student Model · read first ✓ understood
The smaller model in knowledge distillation learning to mimic the teacher's behavior.

Nearby papers

Frontier · #7 Depth Anything 3: Recovering the Visual Space from Any Views A simpler recipe beats specialised 3D pipelines, suggesting 3D perception can ride on general vision backbones. Frontier · #15 Seedream 4.0: Toward Next-generation Multimodal Image Generation It set the bar for commercial image generation and editing in one fast system. Frontier · #17 Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer It shows state-of-the-art image generation does not require “scale at all costs”. Frontier · #29 Self-Forcing++: Towards Minute-Scale High-Quality Video Generation It shows a way to long video without long training videos or long-video teachers.