Kidney endoscopy video to preoperative CT alignment for depth estimation

Monocular depth estimation from kidney endoscopy optimally requires ground truth depth information for endoscopy videos in order to train a neural network. Kidney endoscopy is not compatible with real-time tracking of the scope tip via, e.g., optical trackers, which makes it difficult to directly obtain such depth information. To address this problem, we propose a pipeline to reconstruct a 3D point cloud from the endoscopy video, which we register to a segmentation obtained from the preoperative CT image. We then use the estimated camera poses from the reconstruction and the segmentation surface geometry from preoperative CT to create synthetic endoscopy frames and accurate depth information. We validate this method on kidney phantoms with sub-millimeter RMSE accuracy (0.61 ± 0.08 mm) and demonstrate it on real patient data (0.86 mm).