A multimodal framework to identify molecular mechanisms driving patient group-associated morphology through the integration of spatial transcriptomics and whole slide imaging

Spatial organization of the disease microenvironment informs patient prognosis. Key modalities for studying spatial biology include H&E images (WSIs) for tissue structure and spatial transcriptomics (ST) for transcriptome-level programs. Spatial analysis aims to (1) identify markers linked to clinical outcome, (2) understand functional programs driving these associations, and (3) guide targeted therapies. Current research addresses these topics but offers limited explainability across the full morphology - molecular mechanism - outcome axis. Further, given the abundance of WSIs and limited availability of ST, there is a need for analyses integrating these complementary datasets. We present an AI-driven framework combining foundation-model features, multiple-instance learning, unsupervised clustering, and molecular analyses to identify mechanisms underlying outcome associated patterns. Applied to HER2+ breast cancer, we identify CCND1 and PTK6 signaling in tumor regions linked to trastuzumab resistance, consistent with prior studies. Our approach offers interpretable insights for multi-level resistance mechanisms, tissue-specific drug targeting, and precision medicine.