Multi-head Block-based Sparse Large Kernel Convolutional Neural Network

In recent years, Convolutional Neural Networks (CNNs) have demonstrated competitive performance against Vision Transformers (ViTs) in vision tasks by leveraging large-kernel depthwise convolution (DWConv). However, the expensive memory access cost of large-kernel DWConv has led to a sharp decline in the speed of popular CNNs. To mitigate the speed loss caused by the large-kernel DWConv, we propose a novel and efficient convolution, termed multi-head block-based convolution (MBConv). In comparison to the H×W window-based dense modeling of large-kernel DWConv, our MBConv employs several H×1 or 1×W independent small blocks for sparse modeling. This innovative technique not only ensures effective local continuity modeling and a large receptive field but also achieves a significant reduction in memory access costs, resulting in a substantial speed improvement. Furthermore, our MBConv incorporates a multihead structure to compensate for partial spatial information loss resulting from sparse modeling. Through the aforementioned design, our model achieves a perfect balance between performance and speed. For instance, the experiments on the ImageNet-1K demonstrate that our MBNeXt-B1 achieves significant improvements in both training throughput (+47%) and inference throughput (+29%) while maintaining a similar top-1 accuracy compared to the well-known ConvNeXt-T.