Principles of Hand-eye Calibration and its Application to Medical Augmented Reality

To implement accurate AR navigation, real-time AR-core transformation from the coordinate system of the camera to that of a virtual object is necessary. If feature points in the camera image and virtual object are detected robustly and paired suitably, the direct calculation of the AR-core transformation is feasible for real-time update. However, in endoscopic or microscopic surgery, determining the AR-core transformation requires feature points within the patient’s body. It is challenging and time-consuming to determine robust feature points owing to constraints such as narrow working space, featureless organ structures, and organ deformation. Consequently, direct calculation can retrieve unreliable results. Thus, a position sensor has been commonly utilized for updating the AR-core transformation indirectly in real time given its accuracy, measurement speed, and reliability. Optical trackers are widely used when implementing AR for surgical applications to compensate for camera or patient movement in real time. A rigid transformation between the image and camera frames is necessary and should be updated for real-time AR. For this purpose, two preoperative steps are essential: image-to-patient registration to represents the transformation between the image and patient frames, and hand–eye calibration to represents the transformation between the camera frame and the frame of an optical tracker marker attached to the camera. We have found an optimal solution to acquire these two transformations in one-step, achieving the high accuracy in AR display.