Acquiring Aligned Endoscopic and Depth Image Pairs Using Structured‐Light Projection, Neural Surfaces and an Electromagnetic Positional Sensor

Minimally invasive endoscopic procedures are gaining importance for better patient outcomes. While traditional systems use standard cameras, there is an increasing demand for 3D information to improve diagnostic and surgical accuracy. As a result, various 3D endoscopic systems have been developed, with structured-light (SL) projection gaining traction for its compatibility with existing equipment. Despite advances in SL based scan, practical 3D endoscopy still faces key challenges. These include issues like pattern interference and pose estimation failures. Additionally, removing projected patterns to obtain clean texture images remain crucial for medical diagnosis. To address these challenges, we propose an approach based on neural signed distance field (neural SDF), incorporating a pattern reflection model. The method integrates (1) camera pose estimation using an electromagnetic sensors, (2) 3D shape measurement via SL projection, (3) multi-frame shape integration using a neural implicit surface representation, and (4) removal of projected patterns from endoscopic images using a pix2pix model. To ensure robust optimization, electromagnetic tracking provides accurate initial camera poses for optimization. By simulating SL projection in the neural surface model and optimizing camera pose, projector pose, and surface geometry, and by removing SL projection from the captured images, we can obtain paired sequences of endoscopic and depth maps.