Modal interaction attention-based multimodal fusion network for early prediction of response to neoadjuvant chemotherapy in breast cancer

Early prediction of patient response to neoadjuvant chemotherapy (NAC) is crucial for developing personalized breast cancer treatment plans. Current research predominantly focuses on single ultrasound modalities, overlooking the potential benefits of integrating multiple modalities. This study aims to develop a multimodal fusion method that emphasizes modality interaction, leveraging deep learning to predict early whether a breast cancer patient will achieve pathologic complete response (pCR) after NAC. Our approach incorporates ultrasound, contrast-enhanced ultrasound, histopathologic images, and clinical features. We propose a multi-modal fusion network, consisting of an encoder to extract multi-sequence and cross-modal features, and a decoder to fuse them. In the encoder, we propose a modal interaction attention mechanism, enabling interaction of brightness-mode ultrasound information along the temporal dimension, and both brightness-mode ultrasound and contrast-enhanced ultrasound data across spatial dimensions via convolutional layers and attention modules. Histopathologic images are incorporated to capture cell-level features. The decoder uses a Transformer architecture to fuse these diverse features, while a self-supervised feature orthogonalization loss aids in learning modality-specific features. Experimental outcomes demonstrate that our method achieves robust predictive performance (ACC=0.860, AUC=0.898), surpassing single-modality ultrasound models, single histological models, and simple feature concatenation methods, thereby validating the effectiveness of our approach. Our experimental results indicate that the modal interaction attention-based network effectively addresses the challenge of multi-modal data fusion. This proposed method holds promise for early prediction of NAC efficacy, enabling the development of more optimal, personalized treatment plans for patients.