FishSegMSLA: Real-Time Semantic Segmentation Using Multi-Scale Linear Attention and Content-Guided Attention for Surround-View Fisheye Images

Semantic segmentation is fundamental and crucial for the visual scene understanding in autonomous driving. Self-driving vehicles need a broader field of vision to accurately sense their surroundings, especially at intersections. Compared to regular images, Fisheye images offer an expanded field of view (FOV). However, the pronounced geometric distortion present complicates modeling, resulting in a sharp decline in the performance of neural networks typically optimized for conventional images. While using large composite models can enhance neural network performance, it also incurs substantial time costs and hardware expenses. Therefore, we have designed a lightweight semantic segmentation network for fisheye images, named FishSegMSLA. We used the mask2former network as a baseline and made several improvements upon it. First, we replaced the original backbone of the mask2former model with MSCAN to reduce model complexity. Second, we employed a multi-scale linear attention mechanism to improve the network's capacity to handle geometric distortions in fisheye images. Finally, we implemented a content-guided attention mechanism for feature fusion, effectively merging features and assisting gradient flow. We performed comprehensive experiments on the WoodScape and FisheyeCityScapes datasets.