3D Visualization from Single-Chip Camera in Single Port Access Fetal Robotic Surgery

Single Port Access (SPA) Spina Bifida surgery faces critical visualization limitations due to the use of constrained monocular fetoscopes, leading to inadequate depth perception. To address this, we introduce a groundbreaking monocular 3D reconstruction pipeline that leverages advanced depth estimation and segmentation models. Our method begins by processing the input image to perform static mesh registration and generate a comprehensive static depth map, which is then simultaneously fed into segmentation and depth estimation modules. Utilizing robot forward kinematics and instrument masks, we accurately calibrate the instrument depth map. The mesh and instrument depth maps are subsequently integrated and reprojected into 3D space. The final step involves mapping the input RGB image to render colorized point clouds, providing a detailed 3D visualization. Unlike existing solutions, our system delivers metric-scale 3D scene reconstruction with minimal latency by optimizing the use of pre-operative tissue meshes and precise instrument kinematics. Experimental results demonstrate a depth accuracy of 3.9 mm and an impressive processing latency of 50 ms, facilitating real-time 3D visualization and enhancing surgical precision.