DSMT: RGB-D Dual-Stream Multi-Task Model for Multi-Granularity Semantic Understanding

Understanding semantic information of objects and scenes from images is an important part of mobile robot perception, providing a visual foundation for its operation in complex environments. This paper proposes a dual-stream multi-task (DSMT) model for realizing three different granularities of semantic understanding simultaneously: object detection, semantic segmentation, and scene classification. By integrating RGB images with geometric information from depth images, our model mitigates the deterioration of feature extraction that typically arise due to similar object textures and ambient lighting variations. Specifically, a Coordinate Attention Fusion Module (CAFM) is designed to mutually refine RGB and depth features. Subsequently, a Dynamic Task Awareness Module (DTAM) is developed to enable dynamic perception at various levels of granularity. Additionally, an RGB-D multi-task dataset in the context of warehouse logistics is constructed, and our model is validated using this dataset. The experimental results demonstrate the effectiveness of the proposed method. Code will be available at: https://github.com/BUCT-IUSRC/ResearchDSMT.