Augmented Reality-Assisted Surgical Guidance System for Kidney Nephrectomy with Remote Expert Consultation

This paper represents an augmented reality(AR)-assisted surgical guide system with a novel teleconsultation-assisted system that enables remote expert consultation during partial nephrectomy operations. The system includes three components: a surgical planning software, a virtual reality-based surgical guidance software and a web- based teleconsultation software. Deep learning based kidney segmentations, along with the annotation of the generated masks, are performed in the surgical planning application using three- dimensional patient imaging. The real-time tracking data from region of interest (ROI) is used for localization with the usage of fiducial markers. Using this positional data, patient images are aligned and registered within the intraoperative space. The teleconsultation component allows remote experts to view the surgeons AR display, communicate via vioce and send annotated images of the surgical site for surgical guidance purposes. The proposed framework aims to bridge the gap between on-site and remote surgical expertise, improving accessibility and efficienct in surgical prodecures.