Fuzzy Knowledge Distillation

Knowledge Distillation (KD) is a technique that transfers knowledge from a teacher model to a student model. KD is designed to reduce the number of model parameters while achieving better performance than training student model alone. This paper introduces a novel fuzzy knowledge distillation method that employs fuzzy clustering algorithms to cluster intermediate layer features of both teacher and student models. The differences between features are evaluated using fuzzy membership vectors. Specifically, the Kullback-Leibler (KL) divergence is used to calculate the distance between the fuzzy membership vectors corresponding to the intermediate layer features of the teacher and student models. This KL term is then integrated into the knowledge distillation framework. Additionally, the updating process of the cluster centers for intermediate layer features is implemented using mini-batch stochastic gradient descent. Experimental results demonstrate that the proposed fuzzy knowledge distillation method efficiently facilitates knowledge transfer and achieves higher accuracy than existing methods.