GlassMolder: Transparent Object Reconstruction With Silhouette-Guided Object-Centric Diffusion

Depth reconstruction for transparent objects is a challenging problem, where surface feature matching methods are hindered by complex refraction and reflection. Existing learning-based reconstruction methods by regressing or completing depth maps for entire scenes are data-costly and lack generalization in different environments. To solve this problem, we propose a novel transparent object reconstruction pipeline with a guided object-centric 3D diffusion model. Specifically, we train an unconditional 3D diffusion model with only 3D point cloud data. To control the output of the diffusion model, we design a silhouette-based guidance function and a completion framework with outline points for each step of diffusion process. Specifically, for each step, we design a re-projection pipeline to estimate a silhouette with uncertainty and constrain the partially-noised point cloud to align with it. We further apply stereo matching to compute the outline points in the stereo silhouettes and use a completion framework to fuse them with the partially-denoised point cloud. Finally, we transform the transparent objects to the world frame by applying the transformation from pose estimation. Experiment results show that our method can achieve state-of-the-art performance for transparent object depth reconstruction compared to existing depth regression and completion methods.