A Twin-network Architecture for RGB-based Panoramic Semantic Segmentation

In panoramic semantic segmentation field, increasing studies have focused on the RGB-D input, to enhance the precision of semantic segmentation. However, acquiring depth information in real-world applications remains a challenge. In this paper, we propose a novel twin-network architecture with a two-stage training approach. our proposed method only requires RGB input, but incorporates depth information within it. First, we train the depth branch to extract depth cues from RGB images. Then, we concatenate the latent depth features with the high-level semantic features, aligning depth and semantic information, serving as the input of segmentation decoder. Furthermore, we propose a technique to address distortion in panoramic images. By embedding Deformable Multilayer Perceptron(DMLP) modules on skip connections, we effectively eliminate distortions in panoramic images while fusing local and global image characteristics. Experimental results validate the exceptional performance of our proposed method on two widely used indoor panoramic datasets.