Compressing Deep Neural Networks with Goal-Specific Pruning and Self-Distillation

Neural network (NN) compression aims at reducing the model size and receives much research attention. Nevertheless, we observe that when compressing convolutional neural networks (CNNs), previous approaches may not well measure the impact of filters to loss, resulting in a significant performance degradation after compression. On the other hand, for compressing the fully connected neural networks (FCNNs), we observe that converting the weight matrix to the block diagonal structure would result in better compression. Therefore, for compressing CNNs, we propose a new pipeline in this article, named Retraining-Aware Pruning (RAP) , with a new self-distillation approach, named High-Level Activation-Guided Attention-Preserving Self-Distillation (HAP) and a novel filter pruning strategy, named Normalized Gradients and Geometric Median (NGGM) to effectively improve the accuracy and reduce the model size. Further, for reducing the model size of FCNNs, we formulate a new research problem, i.e., Compression with Difference-Minimized Block Diagonal Structure (COMIS) , and propose a new algorithm, Memory-Efficient and Structure-Aware Compression (MESA) to effectively prune the weights into a block diagonal structure to significantly boost the compression rate. Extensive experiments on different models show that our approaches significantly outperform the state-of-the-art baselines in terms of compression rate, accuracy, and inference speed-up.