Depth Completion and Inpainting for Specular Objects

ABSTRACT Depth images or point clouds offer true three‐dimensional insights into scene geometry, making depth perception essential for downstream tasks in computer vision. However, current commercial depth sensors often produce dense estimations with lower accuracy, especially on specular surfaces, leading to noisy and incomplete data. To address this challenge, we propose a novel framework based on latent diffusion models conditioned on RGBD images and semantic labels for depth completion and inpainting, effectively restoring depth values for both visible and occluded parts of specular objects. We enhance geometric guidance by designing various visual descriptors as conditions and introduce channel and spatial attention mechanisms in the conditional encoder to improve multi‐modal feature fusion. Using the MP6D dataset, we render complete and dense depth images for benchmarking, enabling a comprehensive evaluation of our method against existing approaches. Extensive experiments demonstrate that our model outperforms previous methods, significantly improving the performance of downstream tasks by incorporating the predicted depth maps restored by our model.