Vision-language model for report generation and outcome prediction in CT pulmonary angiogram

Accurate and comprehensive interpretation of pulmonary embolism (PE) from Computed Tomography Pulmonary Angiography (CTPA) scans remains a clinical challenge due to the limited specificity and structure of existing AI tools. We propose an agent-based framework that integrates Vision-Language Models (VLMs) for detecting 32 PE-related abnormalities and Large Language Models (LLMs) for structured report generation. Trained on over 69,000 CTPA studies from 24,890 patients across Brown University Health (BUH), Johns Hopkins University (JHU), and the INSPECT dataset from Stanford, the model demonstrates strong performance in abnormality classification and report generation. For abnormality classification, it achieved AUROC scores of 0.788 (BUH), 0.754 (INSPECT), and 0.710 (JHU), with corresponding BERT-F1 scores of 0.891, 0.829, and 0.842. The abnormality-guided reporting strategy consistently outperformed the organ-based and holistic captioning baselines. For survival prediction, a multimodal fusion model that incorporates imaging, clinical variables, diagnostic outputs, and generated reports achieved concordance indices of 0.863 (BUH) and 0.731 (JHU), outperforming traditional PESI scores. This framework provides a clinically meaningful and interpretable solution for end-to-end PE diagnosis, structured reporting, and outcome prediction.

Authors

Institutions

Publication Details

Journal
npj Digital Medicine
Published
2025-07-12
DOI
https://doi.org/10.1038/s41746-025-01807-8
Citations
13
Primary Topic
Venous Thromboembolism Diagnosis and Management
Type
article
Field-Weighted Citation Impact
24.31

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Vision-language model for report generation and outcome prediction in CT pulmonary angiogram

Colin F. Greineder, Zhusi Zhong, Grayson L. Baird, Cheng Ting Lin et al.
13 citations
npj Digital Medicine
Venous Thromboembolism Diagnosis and Management
24.31
article

Vision-language model for report generation and outcome prediction in CT pulmonary angiogram

Colin F. Greineder, Zhusi Zhong, Grayson L. Baird, Cheng Ting Lin, V. Somasundaram, Scott Collins, Zhicheng Jiao, Xue Feng, Ihab R. Kamel, Michael K. Atalay, Yuli Wang, Shreyas Kulkarni, Harrison X. Bai, Jing Wu, Sun Ho Ahn, Wen-Chi Hsu, Zhuoqi Ma, Lulu Bi
article en
13 citations

Abstract

Accurate and comprehensive interpretation of pulmonary embolism (PE) from Computed Tomography Pulmonary Angiography (CTPA) scans remains a clinical challenge due to the limited specificity and structure of existing AI tools. We propose an agent-based framework that integrates Vision-Language Models (VLMs) for detecting 32 PE-related abnormalities and Large Language Models (LLMs) for structured report generation. Trained on over 69,000 CTPA studies from 24,890 patients across Brown University Health (BUH), Johns Hopkins University (JHU), and the INSPECT dataset from Stanford, the model demonstrates strong performance in abnormality classification and report generation. For abnormality classification, it achieved AUROC scores of 0.788 (BUH), 0.754 (INSPECT), and 0.710 (JHU), with corresponding BERT-F1 scores of 0.891, 0.829, and 0.842. The abnormality-guided reporting strategy consistently outperformed the organ-based and holistic captioning baselines. For survival prediction, a multimodal fusion model that incorporates imaging, clinical variables, diagnostic outputs, and generated reports achieved concordance indices of 0.863 (BUH) and 0.731 (JHU), outperforming traditional PESI scores. This framework provides a clinically meaningful and interpretable solution for end-to-end PE diagnosis, structured reporting, and outcome prediction.

npj Digital MedicineVol. 8(1)
Central South University (CN), Johns Hopkins University (US), Brown University (US), University of Michigan (US), Johns Hopkins Medicine (US), Chang Gung Memorial Hospital (TW), Retina Associates of Kentucky (US), Michigan Medicine (US), Linkou Chang Gung Memorial Hospital (TW), Second Xiangya Hospital of Central South University (CN), University of Colorado Denver (US)
Johns Hopkins University, National Institutes of Health, National Cancer Institute
Openalex Percentile: Top 0%
Venous Thromboembolism Diagnosis and Management
24.31
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.