Unsupervised depth completion based on RGB image and sparse depth map

Depth map generated by LiDAR is highly sparse and irregularly distributed. Subsequent applications like power transmission lines monitoring need accurate depth completion algorithms based on depth and RGB images. Existing supervised methods are trained using the semi-sparse ground truth in the KITTI dataset and these methods have poor generalization, besides, almost all of the existing unsupervised methods use photometric consistency loss for training, which is fragile and unstable. This paper proposes a new multi-modal unsupervised depth completion method. We use the RGB-D distribution consistency constraint to supervise the completion of real sparse depth map without using ground truth, then we model the depth completion problem as an optimal transfer problem and show that it can be trained using GAN. We define a reasonable loss function to balance the domain gap between real and simulated data, and design a two-channel generator and a discriminator. We conducted experiments using KITTI and VKITT dataset, and the results show that our method reached SOTA in RMSE and MAE.