On entropic sparsification of neural networks

Selecting the optimal architectures for neural networks (NNs) in general is a NP-hard problem - and is computationally very challenging in practical applications. Here we explore the idea of data-driven entropic relaxation of this problem, learning the sparse sub-network from the full (dense) pretrained NN. On a set of benchmark problems we show that deploying the entropic sparsification algorithm allows a numericaly-scalable identification of sparse sub-networks with a sublinear scaling of the cost: (i) on the MNIST dataset proposed network sparsification approach removes 80 − 92 % of weights of the baseline network model with around 0.2% of accuracy loss only, and with 6.5- to 8.5-fold speedup in computational runtime; (ii) for the Swiss double-roll problem challenge, even higher sparsification rates ( > 98 % ) of the baseline model are achieved, without any degradation in the performance of the obtained sparse network models. • Neural networks are overparameterized: same performance achievable by smaller models. • SPARTAn sparsifies linear layers of deep networks using entropic regularization. • Entropy minimization enforces sparsity on network and serves as sparsity constraint. • SPARTAn enables scalable identification of sparse subnetworks with sublinear cost. • More than 80% of weights can be removed without loss of accuracy on 2 datasets.