Depth Enhancement Mask Mapping Network With Multi-Teacher Distillation for RGB-D Scene Parsing

The proposed method in the study, called DEMMNet-KD, aims to address the challenges of scene parsing in RGB-D indoor scenes. The study recognizes the importance of lightweight models for pixel-intensive tasks like scene parsing, and proposes a depth enhancement mask mapping network (DEMMNet) with multi-teacher knowledge distillation approach to achieve both reduced model size and high accuracy. DEMMNet introduces a new segmentation paradigm called class-position separate segmentation to reduce the difficulty of delineating and classifying the segmentation region. It also includes a depth spatial enhancement module based on the attention mechanism to effectively utilize depth information in a single-stream network design. To improve performance without increasing complexity, DEMMNet-KD utilizes Mix Transformer encoder (MiT) as a backbone and employs multi-teacher knowledge distillation. This allows semantic knowledge to be transferred from large teacher networks with different performances to the student network. The experiments conducted on NYUDv2 and SUN RGB-D datasets demonstrate that DEMMNet-KD achieves state-of-the-art performance with fewer additional learnable parameters and minimal computational effort. Utilizing MiT-B2 as the backbone network, the DEMMNet-KD method attained mean Intersection over Union (mIoU) scores of 54.71% and 51.03% on the two datasets, respectively, thereby achieving an optimal balance between accuracy and efficiency. Furthermore, the scalability of this method has been demonstrated across various domains and modalities. Overall, the proposed DEMMNet-KD approach provides an efficient solution for scene parsing in RGB-D indoor scenes, enhancing the capabilities of bionic binocular robots in perceiving their environments and serving human society effectively. The code is available at https://github.com/SHARKALAKALA/DEMMNet.