Pruning-Based Optimization of Model Distillation Algorithm Using Explainable Machine Learning Methods

Model distillation, as an important model compression technique, typically focuses only on performance improvement while neglecting the interpretability analysis and refinement of the teacher model, thereby lacking in-depth exploration of the semantic information within the model. This study proposes a pruning technique based on explainable machine learning methods to optimize the model distillation algorithm. First, a teacher network is trained, and the importance scores of the neurons are calculated using explainable algorithms. Subsequently, the teacher network is pruned according to a set sparsity rate, retaining the neurons with high importance scores. The pruned teacher network is then used for knowledge distillation, effectively transferring its knowledge to the student network. Experimental results demonstrate that this method achieves superior model distillation effects on mainstream neural network models and general datasets, significantly improving model accuracy and enhancing interpretability in the knowledge distillation process.