DREAM: Dynamic Responsibility and Evolutionary Adaptive Modularization for Knowledge Distillation

We introduce novel state-of-the-art methods to enhance knowledge distillation in deep learning, focusing on a “teacher-in-the-loop” approach with iterative incremental knowledge transfer. Building on existing KD frameworks (Hinton et. al. [1], Mirzadeh et. al. [2]), we introduce the DREAM framework. This novel distillation system consists of 1. a dynamic weighting scheme that integrates outputs from both the teacher and intermediate Teacher Assistant (TA) models, and 2. an incremental section-wise bottom-up approach that distills the model section by section. Experiments using ResNet architectures trained on Tiny ImageNet support our techniques by showing improved student model accuracy over existing methods.