Automatic Virtual‐to‐real Calibration and Dynamic Registration of Deformable Tissue for Endoscopic Submucosal Dissection

During endoscopic submucosal dissection, precise and intuitive sensing of target tissues enhances surgical accuracy. Augmented reality (AR) technology currently offers a solution to provide intuitive guidance. To enhance the AR user experience, an automated method for calibrating and dynamically registering deformable tissues is proposed. First, an automatic calibration method is proposed to help register the target tissue from the virtual to the real world. The calibration method is based on a 6D pose estimator, which is built on the feature‐matching network, SuperGlue and the depth estimation network, Metric3D. Subsequently, a dynamic registration method is proposed to track the deformation of the target tissue in real‐time. Moreover, a piece of cloth is utilized for four automatic calibration trials, resulting in a mean absolute error (MAE) of calibration accuracy at 3.79 ± 0.64 mm. The dynamic registration accuracy is also assessed by varying the deformation of the target, yielding an MAE of 6.03 ± 0.96 mm. Finally, an ex vivo experiment involving a piece of small intestine is conducted to validate the effectiveness of the proposed system, with an MAE of 3.11 ± 0.56 mm for AR calibration and 3.20 ± 1.96 mm for dynamic registration error.