Multimodal Deep Learning-Based Classification of Breast Non-Mass Lesions Using Gray Scale and Color Doppler Ultrasound

<b>Objectives:</b> To propose a multimodal deep learning method for the classification of benign and malignant breast non-mass lesions (NMLs) using grayscale and color Doppler ultrasound and to compare the performance of multi-modality and single-modality breast ultrasound (BUS) models. <b>Methods:</b> This retrospective study collected 248 pathologically confirmed NMLs from 241 female patients comprising grayscale and color Doppler BUS images from March 2018 to November 2024. Three types of convolutional neural networks (CNNs), including ResNet50, ResNet18, and VGG16, were evaluated as single-modality (grayscale or color Doppler) models via five-fold cross-validations. The optimal model for each single-modality approach was chosen as the backbone network for multimodal deep learning. Features extracted from grayscale and color Doppler BUS images were then concatenated to predict the probabilities of benignity and malignancy. The diagnostic efficacy of the multi-modality BUS models was comparatively evaluated against single-modality counterparts. <b>Results:</b> The single-modality VGG16 models outperformed the other two CNN types for both grayscale and color Doppler BUS using five-fold cross-validations. Additionally, single-modality grayscale models outperformed single-modality color Doppler models. With a mean accuracy of 91.54%, sensitivity of 94.15%, specificity of 87.30%, F1 score of 0.93, and area under the receiver operating characteristic curve (AUC) of 0.96, the multimodal VGG16 models performed better than single-modality counterparts. <b>Conclusions:</b> VGG 16-based multimodal ultrasound deep learning showed excellent diagnostic efficacy in distinguishing between benign and malignant NMLs, indicating therapeutic potential to help radiologists assess NMLs.