SurgEM: A Vision-Based Surgery Environment Modeling Framework for Constructing a Digital Twin Toward Autonomous Soft Tissue Manipulation

Autonomous soft tissue manipulation in robotic surgery remains challenging. Modeling the tool-tissue interaction, analyzing the tissue structural deformation and monitoring the biomechanical status during surgery manipulation may benefit the development of autonomous surgery; however, there are currently inadequate studies. We propose a vision-based surgery environment modeling framework to simultaneously reconstruct and track the forceps and the tissue, leveraging model-based pose estimation and scene flow-based mesh optimization. We also propose a digital twin based on the framework for continuously modeling the tool-tissue interaction and monitoring the deformation and strain of the tissue surface. Quantitative and qualitative <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ex vivo</i> experiments were conducted to evaluate the proposed system from various perspectives. Results show that the deformation recovery accuracy is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\bm {0.38\pm 0.30}$</tex-math></inline-formula> mm with robustness to occlusion; the instrument pose estimation accuracy is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\bm {0.85\pm 0.57}$</tex-math></inline-formula> degree in rotation and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\bm {2.09\pm 1.41}$</tex-math></inline-formula> mm in translation. The relative positioning between the tissue and the forceps can be correctly modeled in terms of contact detection. The system also correctly reveals the differences in strain distributions in two types of tool-tissue interaction, palpation and traction. With the proposed system, surgical robot systems with the perception of tissue deformation can be developed in the future for optimized and autonomous surgery.