Knowledge Distillation Cheat Sheet

Updated 2026-05-25

🧠Study flashcards on this topic90 cards · spaced repetition→

Knowledge distillation is a model compression technique in machine learning where knowledge from a large, complex teacher model is transferred to a smaller, more efficient student model. Originally introduced by Geoffrey Hinton in 2015, this approach enables deploying powerful AI capabilities on resource-constrained devices while maintaining competitive performance. The core insight is that the soft probability distributions (dark knowledge) produced by teacher models contain rich inter-class relationships that are lost when using only hard labels. In 2025–2026, distillation became a production-critical strategy, with landmark results like DeepSeek-R1 demonstrating that reasoning capabilities can be distilled into models achieving frontier performance at a fraction of the cost; key emerging distinctions include white-box vs. black-box teacher access and on-policy vs. off-policy training, which determine what knowledge can be transferred and how effectively.

What This Cheat Sheet Covers

This topic spans 15 focused tables and 104 indexed concepts, 90 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Core ConceptsTable 2: Distillation CategoriesTable 3: Training ParadigmsTable 4: Loss FunctionsTable 5: Teacher-Student ArchitecturesTable 6: Advanced TechniquesTable 7: Domain-Specific ApplicationsTable 8: Cross-Modal and Specialized MethodsTable 9: Evaluation and MetricsTable 10: Implementation ConsiderationsTable 11: Best PracticesTable 12: Common Pitfalls and ChallengesTable 13: Integration with Other Compression TechniquesTable 14: Theoretical FoundationsTable 15: Emerging Research Directions

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Core Concepts

These foundational terms define the vocabulary of knowledge distillation; understanding them — especially the difference between soft and hard targets — is prerequisite to everything that follows.

Concept	Example	Description
Knowledge Distillation	`teacher = ResNet50` `student = MobileNet` `student.train(teacher.outputs)`	• Training a compact student model to mimic a larger teacher model's behavior • transfers learned representations rather than training from scratch
Dark Knowledge	Teacher outputs: `[0.89, 0.08, 0.02, 0.01]` vs. hard label: `[1, 0, 0, 0]`	Information encoded in the full probability distribution across all classes, revealing inter-class similarities that hard labels cannot capture
Soft Targets	`soft_probs = softmax(logits / T)` `T = 3` → `[0.65, 0.20, 0.10, 0.05]`	Smoothed probability distributions from the teacher that provide richer training signals than one-hot encoded hard labels
Teacher Model	`teacher = BERT-Large` `(340M parameters, 24 layers)`	• Large, high-capacity model with strong performance • serves as the knowledge source for distillation
Student Model	`student = DistilBERT` `(66M parameters, 6 layers)`	• Smaller, efficient model trained to replicate teacher behavior • achieves faster inference with reduced memory footprint
Temperature Scaling	`T = 1` → sharp: `[0.95, 0.03, 0.01, 0.01]` `T = 5` → soft: `[0.60, 0.20, 0.12, 0.08]`	• Dividing logits by temperature T before softmax to control probability smoothness • higher T produces softer distributions that better reveal similarities
Compression Ratio	Teacher: 100M params → Student: 25M params Ratio: `100/25 = 4x` compression	• Ratio of teacher to student model size • typical ML distillation achieves 2-10x compression, LLM distillation can reach 20-50x

Table 2: Distillation Categories

The three foundational categories — response-based, feature-based, and relation-based — describe what is transferred; orthogonal distinctions like white-box vs. black-box and token-level vs. sequence-level describe how and at what granularity the transfer occurs.

Type	Example	Description
Response-based Distillation	`loss = KL_div(student_logits, teacher_logits)`	• Student learns from teacher's final output layer (logits or probabilities) • most common and straightforward approach

The rest of this section is part of the full cheat sheet

Table 3: Training Paradigms

How and when the teacher interacts with the student during training determines the quality of knowledge transfer. The critical emerging distinction is on-policy vs. off-policy: off-policy students never see their own errors, which leads to compounding mistakes at inference time.

Paradigm	Example	Description
Offline Distillation	`# Phase 1: train teacher` `teacher.train(dataset)` `# Phase 2: freeze & distill` `student.train(teacher.predict(dataset))`	• Teacher is fully pre-trained and frozen before student training begins (synonymous with off-policy in LLM literature) • most common setup; static teacher data risks exposure bias

The rest of this section is part of the full cheat sheet

Table 4: Loss Functions

Choosing the right distillation loss shapes what information is transferred. Forward KL is mean-seeking and standard; reverse KL is mode-seeking and better suited for generative LLMs; combined losses balance teacher guidance with ground truth.

Function	Example	Description
KL Divergence Loss	$\mathcal{L}{KL} = \sum{i} p_T(i) \log \frac{p_T(i)}{p_S(i)}$	• Measures divergence between probability distributions (forward KL; mean-seeking) • standard distillation loss that equals zero when student perfectly matches teacher

The rest of this section is part of the full cheat sheet

Table 1: Core Concepts

Concept	Example	Description
Knowledge Distillation	`teacher = ResNet50` `student = MobileNet` `student.train(teacher.outputs)`	• Training a compact student model to mimic a larger teacher model's behavior • transfers learned representations rather than training from scratch
Dark Knowledge	Teacher outputs: `[0.89, 0.08, 0.02, 0.01]` vs. hard label: `[1, 0, 0, 0]`	Information encoded in the full probability distribution across all classes, revealing inter-class similarities that hard labels cannot capture
Soft Targets	`soft_probs = softmax(logits / T)` `T = 3` → `[0.65, 0.20, 0.10, 0.05]`	Smoothed probability distributions from the teacher that provide richer training signals than one-hot encoded hard labels
Teacher Model	`teacher = BERT-Large` `(340M parameters, 24 layers)`	• Large, high-capacity model with strong performance • serves as the knowledge source for distillation
Student Model	`student = DistilBERT` `(66M parameters, 6 layers)`	• Smaller, efficient model trained to replicate teacher behavior • achieves faster inference with reduced memory footprint
Temperature Scaling	`T = 1` → sharp: `[0.95, 0.03, 0.01, 0.01]` `T = 5` → soft: `[0.60, 0.20, 0.12, 0.08]`	• Dividing logits by temperature T before softmax to control probability smoothness • higher T produces softer distributions that better reveal similarities
Compression Ratio	Teacher: 100M params → Student: 25M params Ratio: `100/25 = 4x` compression	• Ratio of teacher to student model size • typical ML distillation achieves 2-10x compression, LLM distillation can reach 20-50x

Table 2: Distillation Categories

Type	Example	Description
Response-based Distillation	`loss = KL_div(student_logits, teacher_logits)`	• Student learns from teacher's final output layer (logits or probabilities) • most common and straightforward approach

Table 3: Training Paradigms

Paradigm	Example	Description
Offline Distillation	`# Phase 1: train teacher` `teacher.train(dataset)` `# Phase 2: freeze & distill` `student.train(teacher.predict(dataset))`	• Teacher is fully pre-trained and frozen before student training begins (synonymous with off-policy in LLM literature) • most common setup; static teacher data risks exposure bias

Table 4: Loss Functions

Function	Example	Description
KL Divergence Loss	$\mathcal{L}{KL} = \sum{i} p_T(i) \log \frac{p_T(i)}{p_S(i)}$	• Measures divergence between probability distributions (forward KL; mean-seeking) • standard distillation loss that equals zero when student perfectly matches teacher