CP Decomposition Initialization Schemes for Speeding-Up of Convolutional Neural Networks

While machine learning and convolutional neural networks (CNNs) are making strides, a persistent effort remains to optimize classification techniques and target redundancies from a large number of parameters naturally present in CNNs. While CNNs have become more accessible across machine learning, the aim is to make their use optimal for central processing unit (CPU) schemes with varying degrees of computational power. Tensor decomposition methods, specifically canonical polyadic (CP) decomposition, look to reduce parameters by compressing specific layers in a CNN. However, they have certain inconsistencies, and their full potential remains untouched as decomposition research is endless, with many facets one can focus on to improve a CNN. We target the initialization method of CP decomposition by introducing a novel decomposition that can handle the complexities often seen with more significant rank and make the starting input matrices more secure. We propose an approach that initializes CP decomposition with our method and then uses network-wide fine-tuning to reduce loss in accuracy. We demonstrate the effectiveness of the proposed model by testing it against CP decomposition with random initialization on a CNN for two different convolutional layers. All comparisons are in terms of the original CNN performance data. We achieve speed-ups of the network’s training time on a CPU by a factor of 3.1× / 2.7× while remaining within a 1% decrease in accuracy compared to the original CNN model. This outperforms the random initialization scheme by a difference of 0.9× / 0.4× in network speed-up.