Application-driven multi-modal depth completion in fringe projection profilometry

• A novel multi-modal depth completion network that merges sparse depth map computed from fringe projection, projector-illuminated grayscale image, and the relative depth map from the Depth Anything v2 Foundation Mode. • A unique metrology approach that addresses the long-standing challenges in fringe projection such as measuring mirror-like reflective surfaces and underexposed regions within a single scan. • A potentially transformative industrial sensing approach for complex and heterogeneous e-waste surfaces. Fringe projection profilometry (FPP), while capable of sub-millimeter accuracy at kilohertz speeds, produces sparse and incomplete depth maps when scanning objects with complex, heterogeneous material properties including specular metallic surfaces, mirror-like reflective regions, and absorptive materials. This is due to measurement failures predominantly in mirror-like reflective regions and underexposed areas where fringe patterns are unreliable or absent. Hard disk drives represent a particularly challenging test case for these limitations, exhibiting all of these problematic surface characteristics within a single assembly. Accurate 3D sensing of such components is critical for automated robotic disassembly in e-waste recycling, where valuable materials such as palladium, aluminum, and the rare earth metal neodymium remain largely unrecovered due to lack of recycling infrastructure. Recent zero-shot depth estimation models, while inaccurate for fine-scale, millimeter-level depth prediction, capture useful geometric priors. In this research, we present a multi-modal fusion approach that combines three data sources: sparse depth map computed from FPP, projector-illuminated grayscale image, and the relative depth map from the Depth Anything v2 Foundation Model. Our lightweight fusion network exploits the lower domain gap in geometric features compared to appearance features, enabling effective learning and sim-to-real transfer with limited synthetic and real-world training data. The network learns to predict dense depth in regions where FPP fails, which is then fused with the original sparse measurements to produce complete depth maps. We demonstrate that this approach achieves a mean absolute error and root mean square error of less than 2 mm on both synthetic and real-world test cases, and critically, achieves good reconstruction fidelity in the sparse regions, paving the way for fine-scale robotic disassembly while avoiding the need for extensive surface treatment or large-scale real-world data collection. Furthermore, our approach addresses the primary limitations of FPP on mirror-like reflective surfaces and underexposed regions within a single scan, and demonstrates a potential roadmap for industrial metrology of parts with similarly challenging optical properties. The code for our multi-modal depth completion network, MMDC-Net, will be publicly available at https://github.com/badri999/MMDC-Net