CNN channel pruning via gradient integration and knowledge distillation

With the advancement of artificial intelligence technology, intelligent applications have permeated all aspects of human life, including automatic driving, intelligent security, and face recognition. Convolutional neural networks (CNNs) are essential models for AI, but their complexity increases with more available data. The complex models are unsuitable for edge devices like wearables and smartphones due to high memory and computational requirements. In this paper, the compact model is achieved by pruning some redundant output channels. To effectively measure the validity of the output channels, this paper proposes a channel pruning algorithm based on gradient integration, which combines the gradient of the channel with the model output and the channel’s values. This pruning could destroy model accuracy, so knowledge distillation is introduced during fine-tuning to leverage the original model’s reasoning ability. Experiments show that our approach improves accuracy by 0.51% while reducing parameters by 30.1% and FLOPs by 26% in the VGG16 model.