Abstract 2429: A multi-modal learning framework to integrate digital pathology image and multi-omics data in breast cancer

Abstract Deep learning has advanced digital pathology and multi-omics data analysis. However, there is still a critical need to integrate these modalities to better understand complex diseases. Deep learning models, such as Prov-Gigapath, a state-of-the-art pathology foundation model, have been developed for analyzing whole-slide images (WSI). Our previously published deep-learning framework, OmicsFootPrint (OFP), transforms high-dimensional multi-omics data into intuitive genomically-arranged circular images and utilizes transfer learning to overcome the challenges of high-dimensional, low-sample-size data. In this study, we aimed to combine WSI and multi-omics data to develop a framework capable of distinguishing between invasive ductal carcinoma (IDC) and invasive lobular carcinoma (ILC), which is clinically significant as it may help explain differential responses to therapies, such as tamoxifen versus letrozole. We enhanced the performance of our OFP model by dividing each multi-omics circular image (1024x1024x3 pixels) into 16 equal-sized patches (256x256x3 pixels). Each patch was trained using a separate EfficientNetV2 architecture with pre-trained ImageNet weights. The probability outcomes from these 16 models were subsequently combined through a meta-learner model using the AutoGluon framework. Concurrently, WSI images corresponding to the same patients were independently trained, maintaining consistent train/validation/test splits. The Prov-Gigapath model, which was trained using a dataset from the Providence Health System, was employed to extract patch-level embeddings from the WSI data. These embeddings were fine-tuned using a LongNet-based masked autoencoder to predict outcomes. Finally, predictions from the OFP model and the WSI-based Prov-Gigapath model were integrated through a meta-learner, preserving the original train/validation/test split. Model performances (AUC) were evaluated using the held-out test set. The gold standard used for the analysis was histological identification of IDC versus ILC, against which the performance of omics data alone, WSI data alone, and the multi-modal integration approach were compared. We evaluated our integrated model as a proof-of-concept using The Cancer Genome Atlas (TCGA) breast cancer cohort, comprising 92 ILC and 391 IDC samples. Multi-omics data, including gene expression, microRNA, copy number variation, and RPPA, were analyzed using the OFP framework, while WSI data were processed using the Prov-Gigapath method. The results demonstrated that the multi-modal integration approach significantly outperformed individual modalities, achieving an AUC of 0.98 compared to 0.87 for OFP (original), 0.95 for OFP (16-patches), and 0.92 for WSI. Integrating WSI and multi-omics data improves predictive accuracy, significantly improving the classification of breast cancer subtypes over histology alone. Citation Format: Naresh Prodduturi, Xiaojia Tang, Hamid R. Tizhoosh, Kevin J. Thompson, Richard Weinshilboum, Karthik V. Giridhar, Judy C. Boughey, Eric W. Klee, Liewei Wang, Matthew P. Goetz, Vera Suman, Krishna R. Kalari. A multi-modal learning framework to integrate digital pathology image and multi-omics data in breast cancer [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 1 (Regular Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_1):Abstract nr 2429.