A Granular Interaction Framework for Surgical Robotic Digital Twins: Enabling Realistic Segment-Level Manipulation in Extended Reality

Abstract Introduction While Robot-Assisted Surgery (RAS) offers significant clinical advantages, adoption is often hindered by steep learning curves. Extended Reality (XR) and Digital Twins (DT) provide promising solutions for surgical training. However, existing integration pipelines require high-fidelity system-level simulations that address complex kinematic constraints. This paper proposes a modernized framework for integrating RAS into Extended Reality (XR) by transitioning from CAD models to scan-derived digital twins (DTs), as well as a force-based interaction paradigm for realistic simulation. Methods Performance profiling was conducted on the Apple Vision Pro (AVP) to establish empirical hardware benchmarks for polygon counts. Utilizing 3D scanning and manual retopology, we developed a digital surrogate of the Dexter Surgical Robotic System, prioritizing model fidelity within stand-alone XR constraints. Within Unity, we evaluated Rigidbody versus ArticulationBody physics for kinematic stability. This necessitated the development of a force-based interaction system supporting both linear feedback (PD) and geometric constraint-based controllers. Results The scan-derived model achieved a spatial accuracy of 1.6mm relative to the physical system. ArticulationBody physics demonstrated superior numerical stability compared to traditional Rigidbody components. While the linear feedback controller showed higher quantitative precision, it required extensive manual gain tuning; conversely, the geometric constraint-based controller provided high-speed stability without iterative calibration. Conclusion This framework modernizes RAS integration by providing a non-CAD-dependent workflow, a prescriptive testing protocol for mesh decimation, and a force-based interaction system that enhances kinematic realism for clinical training.