Extended Reality With HMD-Assisted Guidance and Console 3D Overlay for Robotic Surgery Remote Mentoring

The concept of remote-guided surgery has garnered significant attention among researchers as a possible solution to transcend geographic barriers and facilitate the integration of medical resources across diverse regions. However, establishing effective communication channels between remote specialists and surgeons poses an ongoing challenge in this field. In this letter, we propose an extended reality (XR) remote mentoring system for robotic surgery. The system employs an optical see-through head-mounted display (OST-HMD) to provide real-time 3D endoscopic visualization and enable remote mentors to offer surgical guidance through natural hand gestures and virtual surgical instruments. Additionally, a console 3D overlay method is introduced to enhance the depth perception of the console surgeon by stereo occlusion-aware overlaying augmented surgical tools onto the real endoscopic video. The system incorporates the assistant's role in the surgical team by transmitting the assistant's view to the console surgeon and remote mentor, enriching the information in the operating room. Latency and accuracy experiments have proved the efficiency of the system. The user study in a simulated surgical task shows promising performance for optimizing user experience and reducing communication costs.