Scale-aware monocular reconstruction via robot kinematics and visual data in neural radiance fields

Aim Scale-aware 3D reconstruction of the surgical scene from a monocular endoscope is important for automatic navigation systems in robot-assisted surgery. However, traditional multi-view stereo methods purely utilize monocular images, which can recover 3D structures arbitrarily scaled with the real world. Current deep learning-based approaches rely on large training data for relative depth estimation and further 3D reconstruction with no scale. Inspired by recently proposed neural radiance fields (NeRF), we present a novel pipeline, KV-EndoNeRF, which explores limited multi-modal data (i.e., robot kinematics, and monocular endoscope) for surgical scene reconstruction with absolute scale. Methods We first extract scale information from robot kinematics data and then integrate it into sparse depth recovered from structure from motion (SfM). Based on the sparse depth supervision, we adapt a monocular depth estimation network to the current surgical scene to obtain scene-specific coarse depth. After adjusting the scale of coarse depth, we use it to guide the optimization of NeRF, resulting in absolute depth estimation. The 3D models of the tissue surface with real scale are recovered by fusing fine depth maps. Results Experimental results on the Stereo Correspondence And Reconstruction of Endoscopic Data (SCARED) demonstrate that KV-EndoNeRF excels in learning an absolute scale from robot kinematics and achieves 3D reconstruction with rich details of surface texture and high accuracy, outperforming other existing reconstruction methods. Conclusion Combining multi-modal image data with NeRF-based optimization represents a potential approach to achieve scale-aware 3D reconstruction of monocular endoscopic scenes.