A Multi-Optical and Mechanical Compensation Robotic Surgery System Based on Augmented Reality for Endoscopic Neurosurgery

Abstract Endoscopic treatment is a widely employed clinical modality for the rapid reduction of cranial pressure in cases of intracerebral hemorrhage (ICH). We are working to make endoscopic surgery simpler and more effective by providing surgeons with robot and augmented reality (AR). This paper proposes an AR-based multi-optical and mechanical compensation robotic surgical system that can precisely match the virtual hematoma region with the anatomical structures seen under a robot-wielded neuroendoscope. The custom mark unified multi-optical localization system enables optical tracking over a wide range of angles and breadths. The neuroendoscope registration employs a multimodal error compensation approach based on an optical tracking system and the kinematic model of the robotic arm. Utilizing innovative methods, we register patients, medical images, optical localizers, and both neuroendoscopic and robotic coordinate systems, thereby enabling dual-view AR tracking through video-stream fusion. In multimodal error compensation for neuroendoscopic registration, the calibration error is 0.37 mm. The system's performance was evaluated through ten phantom experiments, showing the registration accuracy of 0.39±0.10 mm, optical localizer AR fusion accuracy of 3.30±1.08%, neuroendoscope AR fusion accuracy of 5.22±1.02%, and optical localizer AR average distance and angle accuracies of 1.73±0.21 mm and 0.38±0.13 deg, respectively. The AR-based robotic surgical system (ARSS) overlays virtual lesions onto the current endoscopic view of the real lesions in real-time, which can help surgeons to localize lesions hidden deep in tissue by an endoscopy. It is anticipated that the ARSS will reduce surgeon fatigue and improve the intuitiveness of endoscopic surgery.