Cross-modality transformer model leveraging DCE-MRI and pathological images for predicting pathological complete response and lymph node metastasis in breast cancer

<i>Objective.</i>Pathological diagnosis remains the gold standard for diagnosing breast cancer and is highly accurate and sensitive, which is crucial for assessing pathological complete response (pCR) and lymph node metastasis (LNM) following neoadjuvant chemotherapy (NACT). Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) is a noninvasive technique that provides detailed morphological and functional insights into tumors. The optimal complementarity of these two modalities, particularly in situations where one is unavailable, and their integration to enhance therapeutic predictions have not been fully explored.<i>Approach.</i>To this end, we propose a cross-modality image transformer (CMIT) model designed for feature synthesis and fusion to predict pCR and LNM in breast cancer. This model enables interaction and integration between the two modalities via a transformer's CA module. A modality information transfer module is developed to produce synthetic pathological image features (sPIFs) from DCE-MRI data and synthetic DCE-MRI features (sMRIs) from pathological images. During training, the model leverages both real and synthetic imaging features to increase the predictive performance. In the prediction phase, the synthetic imaging features are fused with the corresponding real imaging feature to make predictions.<i>Main results.</i>The experimental results demonstrate that the proposed CMIT model, which integrates DCE-MRI with sPIFs or histopathological images with sMRI, outperforms (with AUCs of 0.809 and 0.852, respectively) the use of MRI or pathological images alone in predicting the pCR to NACT. Similar improvements were observed in LNM prediction. For LNM prediction, the DCE-MRI model's performance improved from an area under the curve (AUC) of 0.637-0.712, while the DCE-MRI-guided histopathological model achieved an AUC of 0.792.<i>Significance.</i>Notably, our proposed model can predict treatment response effectively via DCE-MRI, regardless of the availability of actual histopathological images.