A Deterministic–Probabilistic Approach to Neural Network Pruning

Modern deep networks are highly over-parameterized. Thus, training and testing such models in various applications are computationally intensive with excessive memory and energy requirements. Network pruning aims to find smaller sub-networks from within these dense networks that do not compromise on the test accuracy. In this article, we present a probabilistic and deterministic pruning methodology which determines the likelihood of retention of the weight parameters by modeling the layer specific distribution of extreme values of the weights. Our method automatically finds the sparsity in each layer, unlike existing pruning techniques which require an explicit input of the sparsity information. Experiments in the present work show that Deterministic-Probabilistic Pruning consistently achieves high sparsity levels, ranging from 65% to over 95%, while maintaining comparable or improved testing accuracy across multiple datasets such as MNIST, CIFAR-10 and Tiny ImageNet, on architectures including VGG-16, ResNet-18, and ResNet-50.