Virtual Teacher Distillation: Enhancing Performance on Edge Smart Ecosystem Devices

Knowledge Distillation is a method that essentially facilitates the transfer of knowledge from a larger, sophisticated model (named Teacher), into a smaller, less complex model(named Student). Multi-teacher distillation introduces multiple teachers to collaboratively guide the learning process of the student. Knowledge distillation does not alter the model's architecture; it simply replaces the original weight files to enhance algorithm performance. This approach effectively addresses the computational limitations of edge recognition devices in the field of smart ecosystems. However, current multi-teacher distillation methods require multiple pretrained teachers under the same dataset, which makes it challenging to test the performance of the methods in large scale datasets. We propose ‘Virtual Teachers' as a solution, which introduces Gaussian noise into the features generated one single teacher, creating new features. These then act as ‘Virtual Teachers' in place of the additional teachers used in multi-teacher methods. Virtual Teacher method has two main advantages: it eliminates the need for multiple pretrained teachers and reduces the inference. Furthermore, we design a ‘plug-and-play’ masking tool, called Virtual Teacher-Mask (VT-MASK) and a Direction-Norm Loss. Virtual Teacher-Mask has the capability to negate the impact of incorrect information from the Virtual Teachers. Our experiments on CIFAR-100 confirm that the exclusive replacement of Additional Teachers with Virtual Teacher does not compromise the efficacy of current multi-teacher methods. Furthermore, we expand our testing to larger datasets like ImageNet, demonstrating the generalizability of our method.