Minimizing Entropy for Training and Quantization (METaQ): A Novel Algorithm for Neural Network Training and Compression

A neural network compression strategy is developed during the training phase by adding a regularization term $\phi(w)$ to the loss function, in order to minimize the entropy of the network weights. $\phi(w)$ results from a nondifferentiable optimization problem that is computationally very complex to solve. However, fortunately, to train the network, it is not necessary to explicitly find $\phi(w)$; instead, it is sufficient to provide its (sub)gradient to standard machine learning tools (such as PyTorch) to guide the training towards a low entropy weight configuration (in addition to achieving good accuracy). This subgradient can be computed using the optimal Lagrange multipliers $\beta^*$ associated with the set of constraints involving the weights $w$ in the problem that defines $\phi(w)$. In this work, we will develop a procedure to determine $\beta^*$ by applying Lagrangian relaxation and optimization techniques, also developing ad hoc methods for certain sub-problems when necessary for efficiency reasons. Once a trained network with low entropy is obtained, the compression strategy culminates in the quantization of the weights. The task of encoding the weights is implemented via well-known compression algorithms that come arbitrarily close to the entropy. The introduction of the term $\phi(w)$ appropriately modeled for the function to be optimized during training is what makes this work innovative. The tests were conducted using the LeNet-5 network on the MNIST dataset, although the strategy is also applicable to larger networks. The achieved results show a $29\times$ compression of LeNet-5 (Compression Ratio 3.43\%) with an accuracy of 99.01\%, making METaQ a strategy comparable to state-of-the-art NN-compression algorithms. The project's source code is freely available at: https://github.com/cardiaa/METaQ