ITCNET: a visual and language pretraining model based on multimodal fusion in predicting the breast cancer subtypes on DCE-MRI

To develop a deep learning model for precise prediction of breast cancer molecular subtypes by integrating dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI), clinical data, and radiology reports. We analyzed a retrospective cohort of 505 patients with pathologically confirmed breast cancer. Several models were developed and evaluated: image-based models (Swin Transformer, DenseNet121, ResNet50), unimodal models based on clinical data and radiology reports (using BERT), and multimodal fusion models. The proposed ITCNET model integrates all three data modalities. The Swin Transformer was the best image-based model. The report-based BERT model surpassed other unimodal approaches. Among bimodal models, fusing reports with clinical data yielded the highest performance. Ultimately, the comprehensive ITCNET model achieved superior accuracy (0.9377) and AUC (0.9935). The ITCNET model demonstrates that multimodal fusion significantly enhances the prediction of breast cancer molecular subtypes, showcasing the potential of visual-language pretraining to advance precision oncology.