Enhancing classification efficiency in capsule networks through windowed routing: tackling gradient vanishing, dynamic routing, and computational complexity challenges

Abstract Capsule networks overcome the two drawbacks of convolutional neural networks: weak rotated object recognition and poor spatial discrimination. However, they still have encountered problems with complex images, including high computational cost and limited accuracy. To address these challenges, this work has developed effective solutions. Specifically, a novel windowed dynamic up-and-down attention routing process is first introduced, which can effectively reduce the computational complexity from quadratic to linear order. A novel deconvolution-based decoder is also used to further reduce the computational complexity. Then, a novel LayerNorm strategy is used to pre-process neuron values in the squash function. This prevents saturation and mitigates the gradient vanishing problem. In addition, a novel gradient-friendly network structure is developed to facilitate the extraction of complex features with deeper networks. Experiments show that our methods are effective and competitive, outperforming existing techniques.