Mathematically-Grounded Multimodal Attention Network for Breast Cancer Prognosis

Breast cancer is the most frequently diagnosed malignancy in women globally, underscoring the necessity for precise prediction of pathologic complete response (pCR) to enhance therapeutic approaches and patient prognosis. Despite the advancements in deep learning for medical imaging, current methodologies often face challenges in integrating multiple imaging modalities effectively and are frequently devoid of robust theoretical frameworks. In this paper, we propose a Radiomics-guided Multimodal Self-Attention Network (RaMA-net), a cutting-edge deep learning architecture tailored for predicting pCR in breast cancer. RaMA-net seamlessly fuses dynamic contrast-enhanced (DCE) magnetic resonance imaging (MRI) with apparent diffusion coefficient (ADC) maps, providing a more comprehensive understanding of tumor characteristics. Utilizing a 3D CNN encoder coupled with a self-attention mechanism, RaMA-net adeptly captures intricate inter-modality dependencies, supported by a mathematically sound foundation. Moreover, the model incorporates radiomics features via a well-established contrastive learning paradigm, enabling it to focus on clinically significant imaging attributes. We rigorously validate RaMA-net’s capabilities through extensive theoretical analyses, including proofs of its expressive capacity, convergence behavior, and the derivation of a novel generalization bound that accounts for feature alignment with radiomics data. Experimental results highlight that RaMA-net surpasses conventional baseline methods in pCR prediction, offering superior interpretability. This contribution lays a solid theoretical basis for radiomics-driven, multimodal deep learning in medical imaging, with potential implications for a broader spectrum of diagnostic applications beyond breast cancer.