Attention-Refined Two-Branch Networks for Real-Time Semantic Segmentation

In real-time demanding scenarios as autonomous driving, real-time semantic segmentation is becoming more and more crucial. BiSeNetV2 has been shown to be an effective model, but its performance in improving speed is limited, especially while maintaining high accuracy. Furthermore, feature map detail loss results from combining high-level semantic and detail information, which is especially crucial for real-time semantic segmentation tasks. In this paper, an efficient Attention Refined Two-Branch Real-Time Semantic Segmentation Network (ARTRNet) is designed to alleviate the above challenges. Specifically, the whole network adopts a two-branch structure: a spatial detail branch and a lightweight dense connectivity context refinement branch, and the lightweight dense connectivity context refinement branch is composed of a novel downsampling module (DSM) and a lightweight dense feature module, which achieves high efficiency in terms of reduced computational cost and model size. In addition, the attention vector of each feature map is computed by residual linking of the Attention Refinement Module (ARM) to highlight the features. A low-resolution context aggregation module (LRCAM) consisting of lightweight Ghost modules is also proposed to enhance the spatial information processing capability of the lightweight densely connected context refinement branch. In the final fusion stage, the Deformed Convolutional Attention Refinement Fusion Module (DCARFM) is proposed, which can enhance the feature expression of the branch and improve the final segmentation results by performing the attention refinement operation on the dual branches separately. Finally, experiments on Cityscape and CamVid datasets show that ARTRNet achieves a good balance between segmentation accuracy and inference speed. On the Cityscapes dataset, we achieved 75.7% mIoU at 132 FPS and 76.9% mIoU at 96 FPS on higher resolution images.