Hybrid Video and Instance Segmentation for Surgical Scene Understanding in Robotic Mitral Valve Surgery

Objective: Artificial intelligence-based surgical scene segmentation is fundamental to surgical scene understanding. However, the optimal segmentation strategy for robotic cardiac surgery remains unknown. We developed and evaluated a hybrid segmentation framework combining Mask R- CNN and Segment Anything Model 2 (SAM2) for surgical scene recognition during the entry phase of robotic mitral valve surgery. Methods: Intraoperative videos from ten robotic mitral valve operations performed at two institutions were analyzed. Seven anatomical structures and surgical object classes (ascending aorta, diaphragm, lung, pericardium, right atrium, robotic arms, and sutures) were manually annotated from 2172 image frames. We created three patient-level training, validation, and test splits after ruling out any split where an object class was completely missing from validation or test. Three segmentation approaches were compared: Mask R-CNN, Mask R-CNN-prompted SAM2 for all seven classes, and a hybrid late-fusion model combining SAM2 for anatomical structures and Mask R-CNN for robotic arms and sutures. Model performance was assessed using pixel accuracy, mean intersection over union (IoU), and Dice score. Results: Across the three patient-level splits, the hybrid late-fusion model achieved the highest average overall segmentation performance, with pixel accuracy of 0.79-0.91, mean IoU of 0.64-0.74, and mean Dice of 0.76-0.84. At the object level, late-fusion Dice scores across the three splits were 0.86-0.96 for aorta, 0.38-0.90 for diaphragm, 0.77-0.92 for lung, 0.48-0.77 for pericardium, 0.68- 0.90 for right atrium, 0.95-0.97 for robotic arms, and 0.66-0.76 for sutures. Mask R-CNN- prompted SAM2 outperformed Mask R-CNN for the aorta in all three splits and for the lung, pericardium, diaphragm, and right atrium in two of three splits, whereas Mask R-CNN was more reliable for sutures, including Dice scores of 0.76 vs 0.21 and 0.66 vs 0.53 in Splits 2 and 3. By combining prompted SAM2 for anatomy with Mask R-CNN for robotic arms and sutures, late fusion achieved the highest mean Dice in two of three splits (0.84 and 0.76) and the highest mean Dice averaged across all three splits; all-object SAM2 propagation performed slightly better in the remaining split (0.81 vs 0.80). Conclusions: A hybrid segmentation framework combining SAM2 and Mask R-CNN achieved the highest overall segmentation performance in two of three patient-level splits and performed comparably to the best-performing model in the remaining split. These findings suggest that object-specific hybrid segmentation strategies may improve surgical scene segmentation during robotic mitral valve surgery.

Authors

Institutions

Publication Details

Journal
medRxiv
Published
2026-11-18
DOI
https://doi.org/10.64898/2026.11.16.26367936
Primary Topic
Medical Image Segmentation Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Hybrid Video and Instance Segmentation for Surgical Scene Understanding in Robotic Mitral Valve Surgery

Sriharsha Talapaneni, Makoto Mori, Viswajit Kandula, Arnar Geirsson et al.
5 links
medRxiv
Medical Image Segmentation Techniques
preprint

Hybrid Video and Instance Segmentation for Surgical Scene Understanding in Robotic Mitral Valve Surgery

Sriharsha Talapaneni, Makoto Mori, Viswajit Kandula, Arnar Geirsson, Marc P. Pelletier, Danial Ahmad, Mina H. Zaky, Christina J. Waldron, Nikhil Bhasin, Ebony Jernigan, Nashita Bhuiyan, Rohit Ravindra Gulve, Zaid Al-Shoha, Nasir Rahim, Heidi Eren
preprint en

Abstract

Objective: Artificial intelligence-based surgical scene segmentation is fundamental to surgical scene understanding. However, the optimal segmentation strategy for robotic cardiac surgery remains unknown. We developed and evaluated a hybrid segmentation framework combining Mask R- CNN and Segment Anything Model 2 (SAM2) for surgical scene recognition during the entry phase of robotic mitral valve surgery. Methods: Intraoperative videos from ten robotic mitral valve operations performed at two institutions were analyzed. Seven anatomical structures and surgical object classes (ascending aorta, diaphragm, lung, pericardium, right atrium, robotic arms, and sutures) were manually annotated from 2172 image frames. We created three patient-level training, validation, and test splits after ruling out any split where an object class was completely missing from validation or test. Three segmentation approaches were compared: Mask R-CNN, Mask R-CNN-prompted SAM2 for all seven classes, and a hybrid late-fusion model combining SAM2 for anatomical structures and Mask R-CNN for robotic arms and sutures. Model performance was assessed using pixel accuracy, mean intersection over union (IoU), and Dice score. Results: Across the three patient-level splits, the hybrid late-fusion model achieved the highest average overall segmentation performance, with pixel accuracy of 0.79-0.91, mean IoU of 0.64-0.74, and mean Dice of 0.76-0.84. At the object level, late-fusion Dice scores across the three splits were 0.86-0.96 for aorta, 0.38-0.90 for diaphragm, 0.77-0.92 for lung, 0.48-0.77 for pericardium, 0.68- 0.90 for right atrium, 0.95-0.97 for robotic arms, and 0.66-0.76 for sutures. Mask R-CNN- prompted SAM2 outperformed Mask R-CNN for the aorta in all three splits and for the lung, pericardium, diaphragm, and right atrium in two of three splits, whereas Mask R-CNN was more reliable for sutures, including Dice scores of 0.76 vs 0.21 and 0.66 vs 0.53 in Splits 2 and 3. By combining prompted SAM2 for anatomy with Mask R-CNN for robotic arms and sutures, late fusion achieved the highest mean Dice in two of three splits (0.84 and 0.76) and the highest mean Dice averaged across all three splits; all-object SAM2 propagation performed slightly better in the remaining split (0.81 vs 0.80). Conclusions: A hybrid segmentation framework combining SAM2 and Mask R-CNN achieved the highest overall segmentation performance in two of three patient-level splits and performed comparably to the best-performing model in the remaining split. These findings suggest that object-specific hybrid segmentation strategies may improve surgical scene segmentation during robotic mitral valve surgery.

medRxiv
Cornell University (US), Columbia University Irving Medical Center (US), Yale University (US), New York University (US), Columbia University (US)
Good health and well-being
Medical Image Segmentation Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon