Attention-Based Dynamic Channel-Wise Pruning of Neural Networks with Knowledge Distillation Fine-Tuning

Neural networks, characterized by large parameter counts and high computational demands, require efficient optimization methods for deployment on resource-constrained edge devices. This paper introduces a novel channel-wise pruning technique that leverages channel-wise attention to selectively prune entire channels based on their output values. By incorporating a specialized loss function, we drive these attention values to zero, effectively removing redundant channels without manual tuning. Additionally, we employ attention value normalization to control the pruning ratio and a knowledge distillation loss function to preserve model accuracy. Our technique demonstrates significant reductions in memory requirements. Applied to pre-trained networks on the CIFAR-10 dataset, our method reduced the parameters of ResNet-56 and VGG-16 by 55% and 70%, respectively, while maintaining high accuracy levels, decreasing only from 94.3% to 91.3% and from 94.14% to 93.65%. This approach not only improves the practicality of deploying powerful neural models on edge devices but also maintains robust accuracy, representing a significant step forward in neural network optimization.