A multimodal knowledge-enhanced whole-slide pathology foundation model

Computational pathology has advanced through foundation models, yet faces challenges in multimodal integration and capturing whole-slide context. Current approaches typically utilize either vision-only or image-caption data, overlooking distinct insights from pathology reports and gene expression profiles. Additionally, most models focus on patch-level analysis, failing to capture comprehensive whole-slide patterns. Here we present mSTAR (Multimodal Self-TAught PRetraining), the pathology foundation model that incorporates three modalities: pathology slides, expert-created reports, and gene expression data, within a unified framework. Our dataset includes 26,169 slide-level modality pairs across 32 cancer types, comprising over 116 million patch images. This approach injects multimodal whole-slide context into patch representations, expanding modeling from single to multiple modalities and from patch-level to slide-level analysis. Across oncological benchmark spanning 97 tasks, mSTAR outperforms previous state-of-the-art models, particularly in molecular prediction and multimodal tasks, revealing that multimodal integration yields greater improvements than simply expanding vision-only datasets.

Authors

Institutions

Publication Details

Journal
Nature Communications
Published
2025-12-12
DOI
https://doi.org/10.1038/s41467-025-66220-x
Citations
10
Primary Topic
AI in cancer detection
Type
article
Field-Weighted Citation Impact
24.81
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

A multimodal knowledge-enhanced whole-slide pathology foundation model

Huangjing Lin, Zhengyu Zhang, Ronald Chan, Jinbang Li et al.
10 citations
Nature Communications
AI in cancer detection
24.81
article

A multimodal knowledge-enhanced whole-slide pathology foundation model

Huangjing Lin, Zhengyu Zhang, Ronald Chan, Jinbang Li, Jiabo Ma, Fengtao Zhou, Xiuming Zhang, Yingxue Xu, Yihui Wang, Xin Wang, Shuhua Yang, Hao Chen, Li Liang, Jiguang Wang, Chenglong Zhao, Anjia Han, Cheng Jin, Huajun Zhou, Zhenhui Li
article en
10 citations

Abstract

Computational pathology has advanced through foundation models, yet faces challenges in multimodal integration and capturing whole-slide context. Current approaches typically utilize either vision-only or image-caption data, overlooking distinct insights from pathology reports and gene expression profiles. Additionally, most models focus on patch-level analysis, failing to capture comprehensive whole-slide patterns. Here we present mSTAR (Multimodal Self-TAught PRetraining), the pathology foundation model that incorporates three modalities: pathology slides, expert-created reports, and gene expression data, within a unified framework. Our dataset includes 26,169 slide-level modality pairs across 32 cancer types, comprising over 116 million patch images. This approach injects multimodal whole-slide context into patch representations, expanding modeling from single to multiple modalities and from patch-level to slide-level analysis. Across oncological benchmark spanning 97 tasks, mSTAR outperforms previous state-of-the-art models, particularly in molecular prediction and multimodal tasks, revealing that multimodal integration yields greater improvements than simply expanding vision-only datasets.

Nature CommunicationsVol. 16(1)
Sun Yat-sen University (CN), Chinese University of Hong Kong (HK), Hong Kong University of Science and Technology (HK), Kunming Medical University (CN), HKUST Shenzhen Research Institute (CN), Key Laboratory of Guangdong Province (CN), Nanfang Hospital (CN), Prince of Wales Hospital (CN), The First Affiliated Hospital, Sun Yat-sen University (CN), Shandong Provincial QianFoShan Hospital (CN), Shandong First Medical University (CN), Southern Medical University (CN), Zhejiang University (CN), University of Hong Kong (HK)
Openalex Percentile: Top 1%
AI in cancer detection
24.81
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.