MSF-UNet: Multi-Scale Fusion UNet for Guided Depth Map Super-Resolution

Since depth maps obtained by commercial depth sensors are usually of low resolution, their applications in various real-world computer vision tasks are limited. Therefore, depth map super-resolution (DSR) is a practical and valuable task, which can upsample low resolution (LR) depth maps to high resolution (HR) space. Among them, guided depth map super-resolution (GDSR) is a hot topic in depth image processing. The main idea is to upsample the low resolution depth map by leveraging the high-frequency information contained in the high-resolution color image of the same scene. Existing guided depth map super-resolution methods are often sensitive to noise and interference especially at high magnification rate, where false details and blurred textures may appear in the generated images, thus degrading the quality of the generated depth map. In addition, the color image in the same scene may have inconsistent and redundant texture information relative to the depth image, which will lead to incorrect guidance for super-resolution. To address these problems, this paper proposes a deep neural network called Multi-Scale Fusion-UNet (MSF-UNet) to improve the quality of guided depth map super-resolution. The proposed network is mainly built based on two novel components, named dense residual attention fusion block (DRAFB) and cross modal attention guidance module (CMAG). By training the model to learn the effective mapping relationship between low resolution depth maps and high resolution RGB images, a more accurate guided depth map super-resolution method is achieved. Quantitative and qualitative experimental results show that it can effectively achieve large scale super-resolution of depth maps, and is superior to the existing methods.