Towards Efficient Capsule Networks Through Approximate Squash Function and Layer-Wise Quantization

Capsule networks (CapsNets) have emerged as a promising architecture for various machine learning tasks due to their ability to capture hierarchical relationships within data. However, this structure has computationally intensive operations, particularly in the squash function, which involves square root calculations. In addition, they consume a lot of memory due to the high number of parameters, which makes them difficult to deploy on resource-constrained devices. In this paper, we take advantages of approximate computing and quantization to improve the efficiency of CapsNets performance. An approximate squash function is proposed using on the Fast Inverse Square Root (FISR) algorithm to accelerate square root operations, offering a remarkable speedup of up to 4 times compared to conventional methods. Additionally, we propose a novel algorithm called Least Sensitive Layer First (LSLF) in order to reduce the memory consumption of CapsNets. LSLF prioritizes aggressive quantization of the most error-tolerant layers while moderately quantizing the least sensitive layers against quantization error. Our experimental results demonstrate the effectiveness LSLF in enhancing the efficiency and performance of CapsNets, paving the way for more scalable and resource-efficient deep learning systems.