RGB-D Image Modeling Method Based on Transformer: RDT

In recent years, the widespread application of Transformers in multi-modal computer vision tasks has brought ncreasing attention to indoor scene image segmentation based on RGB-D images. Due to the significant differences in the feature types between RGB images and depth images, multi-modal image fusion has become a key area of research. To address the issue of internal interference in RGB-D images, this paper proposes a new Transformer-based RGB-D image modeling method, named RDT. RDT models RGB and RGB-D images separately by utilizing a Channel Transformer and a Spatial Transformer, effectively utilizing the auxiliary information from depth images while minimizing their interference. RDT aims to enhance performance through a simple approach and can be easily integrated into existing models. Experiments on the NYU Depth v2 dataset demonstrate that the enhanced model incorporating RDT achieves a 0.37% improvement in mIoU and faster convergence, and fewer parameters.