Performer: A High-Performance Global-Local Model-Augmented With Dual Network Interaction Mechanism

In deep learning, convolutional neural networks (CNNs) focus on local information through convolutional kernels, while transformers attend to global information using self-attention mechanisms. The union of these distinct approaches enables a more comprehensive extraction of image features. However, the feature map dimensions of CNN and Transformer differ, leading to dimension mismatch issues when combining these architectures. Additionally, the parameter size of the hybrid model integrating both architectures remains large, making it difficult to train. To further augmenting the interpretation of complex image patterns, we present Performer, a dual-network architecture that seamlessly combines CNNs and transformers, resulting in a novel and efficient representation learning model. In the Performer model, we innovate by devising a unique interaction methodology for CNN and transformer architectures to enhance the image feature extraction capabilities mutually. To counteract issue of dimensionality mismatch, we also introduce a refined transformer block, a advancement over the transformer block of ViT. To validate the effectiveness of Performer, we conduct extensive experiments on both classification and segmentation tasks. Performer achieve an accuracy of 83.37% on the ImageNet-200 dataset. For semantic segmentation, Performer excels on the CamVid and Hippocampus datasets. On CamVid, our model achieves a mean intersection over union (mIoU) of 63.27% and pixel accuracy of 92.11%, demonstrating superior performance in capturing fine details and handling complex scenes effectively. The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/hlf-thh/Performer</uri>.