Learnable attention driven structured compression technique for neural networks

The recent involvement of neural networks in modern applications demands high accuracy even in resource constraint environments. Conventionally, high accuracy requires more computation, memory and energy, especially for models that perform substantially complex tasks. In this paper, we introduce a powerful compression technique named, “Attention Driven Structured Compression (ADSC)” that has the ability to significantly reduce model size while preserving accuracy. ADSC uses learnable attention-inspired MLP to compute filter importance using lightweight neural network making it computationally efficient unlike other compression techniques that require intensive retraining. Furthermore, Instead of relying on fixed parameters, ADSC learns to pick the right filters based on how the patterns actually activate. We have conducted numerous experiments on diverse datasets including MNIST, CIFAR 10/100 and California Housing to demonstrate the effectiveness of ADSC. The ADSC attains 50-60% compression with minimal performance degradation on all the tested datasets. The performance of ADSC is compared with other popular compression methods like L1 Channel Pruning, Lottery Ticket and NISP (Neuron Importance Score Propagation) based on compression ratios, accuracy loss and computational overhead. These comparisons showed that ADSC outperformed existing techniques. It maintains high accuracy even under heavy compression and keeps a balance that is crucial for deploying deep neural networks on mobile hardware.