Singular Value Decomposition for the Efficient Design of Neural Networks

With advances in computational power and the development of new algorithmic approaches, machine learning models have become in-creasingly large and complex. The undoubted benefits of such models are realised at the expense of the extensive computational re-sources used by large datasets, lengthy training times, and the implementation of trained models. The sustainable use of such computational resources is now being questioned. The work described in this paper aims to better understand the potential for designing machine learning models with maximum efficiency, both in terms of training and of implementation. The fundamental basis of the work is in the singular value decomposition (SVD) of weight matrices in neural network architectures. This decomposition provides a rational basis for disposing of unnecessary information accumulated in the network during training. Whilst some authors have previously made use of the SVD, the novel work described here enables the repeated application of the SVD during network training. This enables the progressive reduction in the dimensions of hidden layers in the network. This in turn enables the right sizing of matrix dimensions as training progresses. The application of the method is illustrated by tackling the signal processing problem of the localisation of a moving sound source. The learning rates, accuracy of localisation, and efficiency of computational implementation are compared for network architectures comprising multilayer perceptrons (MLPs) and recurrent neural networks (RNNs), both architectures having either real or complex elements. The results presented show that the design of networks based on the progressive application of the SVD during training can drastically reduce the training time and the computational requirements of all such models with little or no loss in performance.