Adaptive Sensor-Image Fusion for Enhanced 3D Pose Estimation in Cardiac Intervention

Precise tracking of surgical instruments and anatomical structures is vital for minimally invasive cardiac surgery to perform safe and effective interventions. However, conventional methods that rely on frames for pose estimation are vulnerable to lighting fluctuations, occlusions, and motion blur, the challenges of real surgical environments. To mitigate these challenges, we propose a multi-modal pose estimation architecture allowing us to fuse image data, which can be processed using models such as Swin Transformer, along with position and orientation sensor data via a gated fusion, as well as a layered multi-head attention module. By adaptively addressing visual and sensor-based spatial details, the framework not only contributes to improved robustness against noise and uncertainty but also pose estimation accuracy. We extensively evaluate the proposed method against image-only and sensoronly baselines and demonstrate its superior performance in estimating both position and orientation. The gated fusion mechanism and multi-head attention structure improve the robustness against missing data and sensor noise, thereby enhancing prediction reliability. Position estimation has very close to perfect accuracy in the R<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> > 0.9997 range along all spatial axes. We see that orientation prediction also shows strong linear correlations, where the R<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> values exceed 0.9975 for all components of the quaternion, which confirms the model’s capability of capturing the rotational behavior. The implications of these results are significant, as the proposed framework offers a robust and accurate approach to surgical pose estimation, enabling more reliable cardiac surgical navigation.