DSDP: Real-Time Asymmetric Dual-Stream Instance Segmentation Embedding Depth-Predictive Architecture for Enhanced Scene Understanding

Instance segmentation can help vehicles or robots enhance their understanding of a scene through the pixel-level segmentation of different objects. However, occlusion and boundary blur, especially in cases with similar colors or textures, are still challenges encountered in real-time robust segmentation tasks. To segment a complete instance boundary, the existing 2D approaches fuse local and abstract semantic features derived from the color domain, which leads to homogeneous semantic information, and efficiently separating different objects is difficult in some cases. To address these complicated scenes, inspired by a human prediction processing strategy, where “the brain fills in missing information in advance to help make better decisions”, this study proposes a real-time asymmetric dual-stream instance segmentation algorithm embedding a depth-predictive architecture that provides the covisible depth information of objects. Furthermore, a cross-domain data fusion method and an enhancement-decoupling loss are designed to complement RGB data by utilizing the rich foreground and boundary details of the predicted depth map. In addition, our model can be fine-tuned to integrate it with real depth domain data provided by different input devices. Extensive experiments conducted on the COCO, OCHuman and CityScapes datasets demonstrate the effectiveness of our method. We further deployed our DSDP method on a UAV platform for validation purposes and qualitatively confirmed its validity.