Abstract LB114: Improved breast cancer prognostication through multimodal fusion of H&E and RNA FISH imaging

Abstract Breast cancer prognostication is essential for tailoring of adjuvant treatment. While molecular assays like OncotypeDX provide valuable prognostic information, their widespread adoption is hindered by high costs, long processing times and limited accessibility. Deep learning models analyzing H&E-stained histology slides have emerged as a promising, cost-effective alternative for predicting recurrence risk. Building on this foundation, we developed a multimodal deep learning approach that combines H&E histology with RNA FISH data to enhance prognostic accuracy in breast cancer. To integrate both modalities, we extracted latent features on a patch-level that were later fused to form a multimodal latent patient representation using a transformer-based neural network architecture. For RNA FISH images, features were extracted from each of the six channels individually. The features served as input to the multimodal AI model, which was trained in a weakly-supervised manner to infer a molecular recurrence risk score [0,100]. To assess the increase in performance, three models were trained: (1) An H&E-based model, (2) an RNA FISH-based model, and (3) a multimodal model, which fused information from both modalities. In our cohort of 641 female early-stage breast cancer patients, we analyzed matched H&E and RNA FISH data spanning all PAM50 subtypes using a multimodal deep learning approach. To validate our AI-based recurrence risk scores, we assessed their correlation with progression-free survival (PFS) over a median follow-up period of 96 months. Risk scores were dichotomized at their median, and multivariable Cox Proportional-Hazard analyses were performed, adjusting for age, nodal status, tumor stage, and chemotherapy administration. The resulting hazard ratios (HR) for PFS were 2.09 (95% CI: 1.36-3.22, p<0.001) for the H&E-only model, 2.18 (95% CI: 1.44-3.30, p<0.001) for the RNA FISH-only model, and 2.31 (95% CI: 1.51-3.53, p<0.001) for the multimodal model. While these findings confirm H&E as a powerful prognostic tool, they also demonstrate that incorporating RNA FISH data provides additional prognostic value, enabling more precise risk stratification. Our study validates the effectiveness of a multimodal deep learning approach that combines H&E and RNA FISH data for breast cancer prognostication. By integrating these complementary imaging modalities, we achieved improved risk assessment of progression-free survival outcomes compared to single-modality analyses. Although the performance improvements were modest, they demonstrate how diverse imaging modalities can capture distinct biological information. The enhanced hazard ratios and survival stratification underscore the clinical value of combining molecular imaging data with traditional histopathological features, suggesting a promising direction for more comprehensive prognostic assessments. Citation Format: Firas Khader, Daniel Truhn, Pavol Cekan, Evan Paul, Jakob Nikolas Kather, Omar S.M. El Nahhas. Improved breast cancer prognostication through multimodal fusion of H&E and RNA FISH imaging [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 2 (Late-Breaking, Clinical Trial, and Invited Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_2):Abstract nr LB114.