Compressing Neural Networks Using Tensor Networks with Exponentially Fewer Variational Parameters

The neural network (NN) designed for challenging machine learning tasks is in general a highly nonlinear mapping that contains numerous variational parameters. The complexity of NNs, if unbounded or unconstrained, might unpredictably cause severe issues including overfitting, loss of generalization power, and excessive cost of hardware. In this study, we propose a general compression scheme that considerably reduces the variational parameters of NNs, regardless of their specific types (linear, convolutional, etc.), by encoding them into deep automatically differentiable tensor networks (ADTNs) that contain exponentially fewer free parameters. The superior compression performance of our scheme is demonstrated on several widely recognized NNs (FC-2, LeNet-5, AlexNet, ZFNet, and VGG-16) and datasets (MNIST, CIFAR-10, and CIFAR-100). For instance, we compress 2 linear layers in VGG-16 with approximately 10 7 parameters to 2 ADTNs with only 424 parameters, improving the testing accuracy on CIFAR-10 from 90.17% to 91.74%. We argue that the deep structure of the ADTN is an essential reason for its remarkable compression performance compared to existing compression schemes, which are mainly based on tensor decompositions/factorization and shallow TNs. Our study suggests that the deep TN is an exceptionally efficient mathematical structure for representing the variational parameters of NNs, one that exhibits better compressibility than the commonly used matrices and multiway arrays.