Disparity-constrained Knowledge Distillation for Cyber-Physical Systems and Edge Devices

Miniscule wearable devices with vast data from Cyber-Physical Systems and Internet of Things face challenges in storage, power, and compute due to limited resources. To adapt the state-of-the-art pre-trained models, models and data need to be processed using compression techniques like Knowledge Distillation for its application on edge devices. Knowledge Distillation is limited by student-teacher learning capabilities and several limitations highlighted in introduction. This paper introduces a novel experimental investigation on the impact of size and structural disparity on the performance of knowledge distillation in wearable devices which are otherwise deprived form using these for the given constraints. It uses 12 models of different sizes and architectures as student and teacher networks to distil knowledge from scratch utilizing image augmentation, random subsampling, and checkpoint restoration of student models while training from differently-abled teacher networks to quantify the best accuracy of distilled models at varying size and structural-disparities. The study shows that selecting models with smaller size and structural disparity can significantly enhance the accuracy of knowledge distilled models. The authors suggest further research to develop a new metric for these disparities, which can help identify potential teachers for students constrained by storage and compute limits. We achieved some significant results on standard benchmark datasets like CIFAR10, FMNIST and MNIST that demonstrate that knowledge-distilled models’ accuracy can be improved by selecting student-teacher models with smaller size and structural disparity and suggests further research to develop a new metric.)