Training & Optimization
Distillation Temperature
A hyperparameter in knowledge distillation controlling how soft the teacher’s outputs are.
This concept is essential for understanding training & optimization and forms a key part of modern AI systems.
Related Concepts
- Knowledge Distillation
- Temperature
- Transfer Learning
Tags
training-optimization knowledge-distillation temperature transfer-learning
Related Terms
Knowledge Distillation
Training a smaller 'student' model to mimic a larger 'teacher' model, transferring knowledge while reducing size.
Temperature
A sampling parameter controlling randomness in generation - lower values make output more deterministic, higher more creative.
Transfer Learning
Leveraging knowledge learned from one task/domain to improve performance on a related task with less data.