Scene adaptation network for visual-thermal urban scene semantic segmentation

As one aspect of an intelligent transportation system, urban scene segmentation is important for understanding and analyzing a scene as part of the operation of an autonomous driving system. Over recent decades, research into semantic segmentation has made great breakthroughs due to the development of deep learning. However, in complex urban scenes, the performance of deep learning models is usually seriously affected, and the issue of how to effectively combine information from different spectra, such as visual images with red, green, and blue (RGB) colors and thermal infrared (TIR) images for urban scene segmentation has become a hot topic. To achieve effective feature extraction and fusion from different spectra while simultaneously considering the characteristics of urban scenes, a scene adaptation network (SAN) is proposed here to dynamically adjust and incorporate the features from RGB and TIR images based on the scene conditions. We employ multi-level operations and a self-attention module to fuse and enhance the semantic features. In addition, to promote the learning of the correlated and complementary information between different spectra, we design a self-supervised learning scheme for the training of our SAN. Through comprehensive experiments on three visual-thermal urban scene semantic segmentation datasets, we demonstrate the effectiveness and superiority of the proposed SAN against other state-of-the-art approaches.