Integrated multimodal image fusion and quantification for advanced visualization in mitral valve surgery

Minimally invasive surgery is the state-of-the-art approach for repairing the mitral valve, which controls the blood flow into the left heart chamber. The surgeons rely on camera and sensor technologies to support visualization, navigation, and measurement. As patients are connected to a cardio-pulmonary bypass, the anatomy is severely deformed by the altered pressure conditions. We developed a technique that combines stereo-endoscopic video with three-dimensional transesophageal echocardiography (3D TEE) to improve anatomic visualization and measurement accuracy during mitral valve repairs. Our methodology includes stereo camera calibration, image segmentation, and 3D model reconstruction. Anatomical landmarks are used to align the imaging modalities. This approach allows the visualization of pre-operatively determined mitral valve properties, e.g., overlaying heat maps in stereo endoscopic data. Our validation results showed high precision and accuracy within an error range of 0.5 ± 0.1 mm. The effectiveness of the heatmap visualization in complex prolapse cases varied. In the future, this approach can also be used to visualize local tissue properties or the optimal positions of implants.