Atlas · GenAI 2026
Knowledge Distillation
Knowledge Distillation
conceptPeak: 2020Model CompressionAI consensus: 2/3
Prerequisites
- hardDeep Learning
Distillation trains a student network to mimic a teacher network — both are neural networks requiring DL understanding
- mediumModel Evaluation
Evaluating distillation quality requires comparing student vs. teacher on meaningful metrics
Recommended reference
Hinton et al. (2015) 'Distilling the Knowledge in a Neural Network' — the foundational paper. Updated: Gu et al. (2024) 'MiniLLM: Knowledge Distillation of LLMs'
Notes from AI deep research
Anthropic Opus
Hinton (2015). Teacher → student. W 2026: distillacja frontier do szybkich lokalnych 8B
Google Deep Think
Zdolności modelu 100B w 8B [G#67]
Related skills
- → is subcategory of: Model Fine-Tuning(3/3)
- → is part of: Edge AI(2/3)
- → is subcategory of: LLM Fine-Tuning(1/3)