An AI Approach to Markerless Augmented Reality in Surgical Robots

This paper examines the integration of markerless augmented reality (AR) within the da Vinci Surgical Robot, utilizing artificial intelligence (AI) for improved precision. The main challenge in creating AR for these systems is the small size (5 mm diameter) of the cameras used. Traditional camera-calibration approaches produce significant errors when used for miniature cameras. Further, the use of external markers can be obstructive and inaccurate in dynamic surgical environments. The study focuses on overcoming these limitations of traditional AR methods by employing advanced neural networks for camera calibration and real-time image processing. We demonstrate the use of a dense neural network to reduce the total projection error by directly learning the mapping of a 3D point to a 2D image plane. The results show a median error of 7 pixels (1.4 mm) when using a neural network, as compared to an error of 50 pixels (10 mm) when using a more traditional approach involving camera calibration and robot kinematics. This approach not only enhances the accuracy of AR for surgical procedures but also offers a more seamless integration with existing robotic platforms. These research findings underscore the potential of AI in revolutionizing AR applications in medical robotics and other teleoperated systems, promising efficient and safer interventions.