Simultaneous Color Registration and Depth Completion of Point Clouds with Curriculum Learning

Dense depth completion is critical for 3D computer vision but remains challenging when depth data are sparse and misaligned with color images due to sensor offsets. We propose a fully convolutional neural network architecture that simultaneously performs depth completion and color image registration, effectively addressing the problem of sparse depth maps and misaligned RGB inputs. Our model is trained with a novel synthetic depth generation strategy that mimics real time-of-flight (ToF) sensor noise and occlusion artifacts, helping to bridge the simulation-to-real gap. In addition, we adopt a staged curriculum learning paradigm that progressively increases task complexity over three training phases, from easy alignment scenarios to full-depth completion with simulated sensor noise. By leveraging shared features between the depth and color tasks, the joint model outperforms separate single-task approaches. At the KITTI Depth Completion benchmark, the proposed approach achieves competitive accuracy while using significantly fewer parameters and achieving faster inference than existing methods, demonstrating its effectiveness and efficiency.