A clinical multimodal vision language foundation model with fine-grain explainability

Despite their considerable promise, many current medical foundation models remain modality specific and lack transparent reasoning, limiting clinical adoption. To address this gap, we introduce Explainable Vision Language Foundation Model for Medicine (EVLF-FM), a multimodal vision language foundation model that combines broad diagnostic capabilities with fine-grained explainability. The development and testing of EVLF-FM encompassed more than 1.3 million total samples from 18 global datasets across 10 imaging modalities related to six clinical specialties: dermatology, hepatology, ophthalmology, pathology, pulmonology and radiology. External validation employed 13,443 independent test samples from 17 additional datasets across 10 imaging modalities. EVLF-FM supports disease diagnosis, visual question answering and pixel-level visual grounding with reasoning capabilities. In internal validation for disease diagnostics, EVLF-FM achieved the highest average accuracy, 0.895, outperforming leading generalist and specialist models. In medical visual grounding, EVLF-FM also achieved reasonable performance across nine modalities with average accuracy at 0.5 intersection over union ([email protected]) of 0.842. External testing further confirmed consistent zero-shot and few-shot performance, with competitive F1 scores despite a smaller model size. A hybrid training strategy combining supervised and visual reinforcement fine-tuning enables EVLF-FM to generate clinically interpretable explanations aligned with visual evidence. Human evaluation of these reasoning outputs using the S.C.O.R.E. framework yielded a high overall mean score of 4.25/5, supporting their clinical interpretability. Current limitations include the need for additional validation across diverse healthcare settings. Prospective clinical studies are required to evaluate the safety, effectiveness and clinical utility of EVLF-FM before routine clinical deployment.

Authors

Institutions

Publication Details

Journal
Nature Medicine
Published
2026-10-29
DOI
https://doi.org/10.1038/s41591-026-04655-5
Primary Topic
Multimodal Machine Learning Applications
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

A clinical multimodal vision language foundation model with fine-grain explainability

Rick Siow Mong Goh, Marcus Ang, Darren Shu Jeng Ting, Laura Gutiérrez et al.
1 links
Nature Medicine
Multimodal Machine Learning Applications
article

A clinical multimodal vision language foundation model with fine-grain explainability

Rick Siow Mong Goh, Marcus Ang, Darren Shu Jeng Ting, Laura Gutiérrez, Iain Beehuat Tan, Damon Wing Kee Wong, Tony Kiat Hon Lim, Brian T. Soetikno, Victor Koh, Arun James Thirunavukarasu, Chrystie Wan Ning Quek, Tin Aung, Zhen Ling Teo, Haoran Cheng, Alessandro Berni, Kanae Fukutsu, Daniel Shu Wei Ting, Soon Thye Lim, Eyal Klang, Nicholas P. Ruthmann, Rahat Husain, Tobias Elze, Jodhbir Singh Mehta, Hiok Hong Chan, Nguyen Tuan Anh Tran, Tien Yin Wong, Darius M. Moshfeghi, Gavin Siew Wei Tan, Marcus Chun Jin Tan, Chee Leong Cheng, Yih Chung Tham, Lionel Tim‐Ee Cheng, Rae Young Kim, Yuhe Ke, Zengxiang Li, Ashley Shuen Ying Hong, Le Dinh Linh, Sumit Sharma, Ching Yu Cheng, Christopher S Nielsen, Yang Bai, Leopold Schmetterer, Kelvin Z. Li, Yang Zhou, Jun Zhou, Nan Liu, Jie Yao, Yong Liu, Leonard Yip
article en

Abstract

Despite their considerable promise, many current medical foundation models remain modality specific and lack transparent reasoning, limiting clinical adoption. To address this gap, we introduce Explainable Vision Language Foundation Model for Medicine (EVLF-FM), a multimodal vision language foundation model that combines broad diagnostic capabilities with fine-grained explainability. The development and testing of EVLF-FM encompassed more than 1.3 million total samples from 18 global datasets across 10 imaging modalities related to six clinical specialties: dermatology, hepatology, ophthalmology, pathology, pulmonology and radiology. External validation employed 13,443 independent test samples from 17 additional datasets across 10 imaging modalities. EVLF-FM supports disease diagnosis, visual question answering and pixel-level visual grounding with reasoning capabilities. In internal validation for disease diagnostics, EVLF-FM achieved the highest average accuracy, 0.895, outperforming leading generalist and specialist models. In medical visual grounding, EVLF-FM also achieved reasonable performance across nine modalities with average accuracy at 0.5 intersection over union ([email protected]) of 0.842. External testing further confirmed consistent zero-shot and few-shot performance, with competitive F1 scores despite a smaller model size. A hybrid training strategy combining supervised and visual reinforcement fine-tuning enables EVLF-FM to generate clinically interpretable explanations aligned with visual evidence. Human evaluation of these reasoning outputs using the S.C.O.R.E. framework yielded a high overall mean score of 4.25/5, supporting their clinical interpretability. Current limitations include the need for additional validation across diverse healthcare settings. Prospective clinical studies are required to evaluate the safety, effectiveness and clinical utility of EVLF-FM before routine clinical deployment.

Nature Medicine
Agency for Science, Technology and Research (SG), Massachusetts Eye and Ear Infirmary (US), Cleveland Clinic (US), Harvard University (US), SingHealth (SG), National University of Singapore (SG), University of Calgary (CA), Nanyang Technological University (SG), Hokkaido University (JP), Singapore General Hospital (SG), Singapore National Eye Center (SG), Cleveland Clinic Lerner College of Medicine (US), Wuhan University (CN), University of Oxford (GB), London School of Hygiene & Tropical Medicine (GB), SingHealth Duke-NUS Academic Medical Centre (SG), Singapore Eye Research Institute (SG), Birmingham and Midland Eye Centre (GB), Fondation Ophtalmologique Adolphe de Rothschild (FR), Duke-NUS Medical School (SG), National Cancer Centre Singapore (SG), Stanford Medicine (US), National University Health System (SG), Beijing Tsinghua Chang Gung Hospital (CN), The Catholic University of Korea Seoul St. Mary's Hospital (KR), National Healthcare Group (SG), Aier Eye Hospital Group (China) (CN), International Centre for Eye Health (GB), SingHealth Community Hospitals (SG), Institute of Advanced Intelligence and Computing (SG), Case Western Reserve University (US), Genome Institute of Singapore (SG), Medical University of Vienna (AT), University of Birmingham (GB), Catholic University of Korea (KR), Stanford University (US), Icahn School of Medicine at Mount Sinai (US), Tsinghua University (CN)
Research to Prevent Blindness, Duke-NUS Medical School, Agency for Science, Technology and Research, National Natural Science Foundation of China, Russian Science Foundation, National Institutes of Health, National Medical Research Council, National Eye Institute
Good health and well-being
Openalex Percentile: Top 47%
Multimodal Machine Learning Applications
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon