Deep Learning-Based Segmentation for Autonomous Robot-Assisted Tumor Localization using Ultrasound B-Mode and Strain Elastography Imaging

Accurate tumor localization is challenging during robot-assisted minimally invasive surgery (RAMIS) when there is insufficient sense of touch and tumors may shift relative to pre-operative images. Intraoperative ultrasound imaging helps address this limitation by providing real-time visualization, while strain elastography (SE) shows tissue stiffness and offers functional insights beyond standard B-mode ultrasound imaging. This paper presents a robotics-based system for autonomous scanning and tumor localization during RAMIS. The system employs the da Vinci® Classic surgical robot with the da Vinci Research Kit (dVRK) and a custom ultrasound transducer instrument, enabling the automated acquisition of both B-mode and SE images of a given tissue sample. A deep neural network-based approach is presented for localizing tumors within the collected images that can be integrated into an autonomous robotic system. A hybrid fusion multimodal deep learning framework is developed based on an attention-enhanced U-Net architecture. It is then compared with both unimodal and multimodal networks using early and late fusion strategies. The dataset is collected using both phantoms and ex vivo liver tissue. The results demonstrate that the multimodal approaches generally outperformed the unimodal approaches. Additionally, the hybrid fusion strategy generally performed better than early or late fusion. The quantitative and qualitative results suggest that the hybrid fusion of features from B-mode and SE imaging can improve tumor segmentation accuracy.