Mixed Reality 6D Object Pose Estimation using Deep Learning for Visual Markerless Surgical Navigation

Modern surgical navigation systems usually use external trackers to estimate tool pose. They recognize and track markers such as ArUco/QR codes or reflective spheres. As a consequence, there is additional equipment in the operating room that must be calibrated prior to each procedure. This paper introduces a novel approach that uses only visual information from a Microsoft HoloLens 2 mixed reality headset to estimate the pose of the object. The proposed method is based on PVNet to determine the position and orientation of objects in RGB images and transforms the parameters into the headset coordinate system by incorporating depth data from the built-in ToF camera. The experiments demonstrated the angle estimation error equal to 1.91<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">◦</sup> ±1.16<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">◦</sup> for orientation angles between 0<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">◦</sup> and 90<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">◦</sup> and the translation estimation error equal to 0.64±0.19 cm for translations between 0 cm and 45 cm. Those errors meet clinical requirements in terms of accuracy and allow to use the system in practical conditions.