Pruning Optimization of Deep Learning Model Based on Generalization Estimation

Existing research has shown that deep learning models contain considerable redundancy, requiring compression optimization to eliminate unnecessary parameters and improve computational efficiency. In model compression through pruning, many existing methods often use loss values to guide the identification of sub-network models. However, low training loss does not necessarily indicate good generalization ability. Consequently, reducing model parameters while preserving strong generalization performance has become a key area of research. Addressing the challenge of ensuring strong generalization performance of submodels after optimization through deep learning model pruning, we propose a channel-structured generalization estimation model. The model architecture incorporates Fully Connected Neural Networks (FCNN) and Gated Recurrent Units (GRU) as modules for channel feature extraction. Additionally, we introduce a differentiable pruning algorithm based on generalization estimation. Experimental results demonstrate the capability of the channel-structured generalization estimation model in predicting the generalization gap via regression. Furthermore, the proposed differentiable pruning algorithm effectively safeguards the generalization capability of sub-models. We conduct pruning experiments on NiN-structured convolutional neural network models on two datasets, CIFAR-10 and SVHN. Comparative experimental results demonstrate the effectiveness of this approach: achieving a 0.72% accuracy improvement over the original model on CIFAR-10; and a 0.12% accuracy improvement over the base model on SVHN while reducing FLOPs by 50.2%, validating the effectiveness of the proposed approach.