Abstract PS3-04-15: Merlin: dissecting therapeutically relevant elements of the oncogenic microenvironment using multi-modal deep learning

Abstract Background: At present, clinical decision making and drug discovery are based on an incomplete picture of tumor biology. Deep learning (DL) has demonstrated the ability to infer oncogenic molecular signatures from data acquired during standard oncologic care (e.g., histopathology [H&E]). To advance the paradigm of DL from a tool that merely predicts outcomes into an engine that uncovers new therapeutic insights, we present MERLIN: Multimodal Evaluation of therapeutic Resistance through deep Learning INtegrating multimodal data. Methods: To develop MERLIN, we curated a dataset of 3111 H&E whole slide images (WSIs) and patient-matched clinico-genomic data from The Cancer Genome Atlas-Breast Cancer dataset and cBioPortal. This dataset was split into training (70%), validation (15%), and testing (15%) sets. Using attention-based multiple instance learning and several multimodal fusion techniques, we trained uni-modal (input: H&E) and multimodal (input: H&E and clinical data) models that quantify a patient’s tumor microenvironmental (TME) phenotypes. The ground truth for training these models was defined by each patient’s bulk RNA-sequencing data and Gene Set Enrichment Analysis protocols. Consistent with our prior work, image patching, feature extraction, and data augmentation strategies were applied during model training. Results: MERLIN achieved robust performance in deriving TME insights using only a patient’s H&E WSI and clinical metadata ( Table 1 ). Using attention maps, MERLIN correctly identified morphology on H&E that governs tumor and immune function. A board-certified pathologist (C.M.P.) verified that high-attention areas, as determined by MERLIN, consistently identified spatial patterns of cell populations (e.g. cluster of lymphocytes) that matched with the phenotype of interest (e.g. T-cell cytotoxicity). Additional validation with tissue-matched immunofluorescence is ongoing. To demonstrate MERLIN’s ability to guide clinical decision making, we curated a multi-institutional dataset of 232 H&E WSIs of core biopsies from stage 2 & 3 triple negative breast cancer patients. Consistent with a known link between tumor-infiltrating lymphocyte activity and checkpoint inhibitor response, we identified correlations between the patient’s likelihood of achieving a pathologic complete response to neoadjuvant Keynote-522 and increased expression of T-cell cytotoxicity (p=0.017) as determined by MERLIN’s analysis of a patient’s H&E WSI. Conclusion: MERLIN has illustrated promise in obtaining TME-derived insights and identifying therapeutically relevant spatial patterns of cell populations from H&E and clinical metadata. We are now applying MERLIN to large datasets of clinically acquired data to identify new insights for cancer therapeutics without known a priori mechanisms of resistance. Citation Format: T. Goncalves, D. Pulido-Arias, C. Perrino, M. Cleveland, E. Gerstner, J. Kalpathy-Cramer, K. Flaherty, V. Zanfagnin, A. Iafrate, S. Sirintrapun, A. Parwani, G. Tozbikian, J. Cardoso, D. Sgroi, C. Bridge, A. E. Kim. Merlin: dissecting therapeutically relevant elements of the oncogenic microenvironment using multi-modal deep learning [abstract]. In: Proceedings of the San Antonio Breast Cancer Symposium 2025; 2025 Dec 9-12; San Antonio, TX. Philadelphia (PA): AACR; Clin Cancer Res 2026;32(4 Suppl):Abstract nr PS3-04-15.