Automated Chest X-ray Report Generation Remains Unsolved

Accurate interpretation of chest radiograph images and generation of narrative reports is essential for patient care but places a heavy burden on radiologists and clinical experts. While AI models for automated report generation show promise, standardized evaluation frameworks remain limited. Here we present the ReXrank Challenge V1.0, a competition in the generation of chest radiograph reports utilizing ReXGradient, the largest test dataset consisting of 10,000 studies across 67 sites. The challenge attracted diverse participants from academic institutions, industry, and independent research teams, resulting in 8 new submissions alongside 16 state-of-the-art models previously benchmarked. Through comprehensive evaluation using multiple metrics, we analyzed model performance across various dimensions: differences between normal and abnormal studies, generalization capabilities across healthcare sites, and error rates in identifying clinical findings. This benchmark reveals that automated chest X-ray report generation remains fundamentally unsolved, with significant performance gaps between normal and abnormal studies, where even top-performing models achieve less than 45% error-free reporting on abnormal cases, and substantial variability across healthcare institutions, indicating that robust, clinically-ready systems require continued development before widespread deployment.

Authors

Institutions

Publication Details

Published
2025-12-01
DOI
https://doi.org/10.1142/9789819824755_0017
Primary Topic
COVID-19 diagnosis using AI
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Automated Chest X-ray Report Generation Remains Unsolved

Pranav Rajpurkar, Ou Huang, Subathra Adithan, Xiaoman Zhang et al.
COVID-19 diagnosis using AI
article

Automated Chest X-ray Report Generation Remains Unsolved

Pranav Rajpurkar, Ou Huang, Subathra Adithan, Xiaoman Zhang, Inhyeok Baek, Luyang Luo, Mercy Ranjit, Aaron Nicolson, Josh Miller, İbrahim Ethem Hamamcı, Zaiqiao Meng, Kenza Bouzid, Hanbin Ko, Shruthi Bannur, Zongwei Zhou, Julián Acosta, Hongyu Zhou, Bevan Koopman, Shaury Srivastav, Xiaoli Yang, Xi Zhang, Sriram Gnana Sambanthan
article en

Abstract

Accurate interpretation of chest radiograph images and generation of narrative reports is essential for patient care but places a heavy burden on radiologists and clinical experts. While AI models for automated report generation show promise, standardized evaluation frameworks remain limited. Here we present the ReXrank Challenge V1.0, a competition in the generation of chest radiograph reports utilizing ReXGradient, the largest test dataset consisting of 10,000 studies across 67 sites. The challenge attracted diverse participants from academic institutions, industry, and independent research teams, resulting in 8 new submissions alongside 16 state-of-the-art models previously benchmarked. Through comprehensive evaluation using multiple metrics, we analyzed model performance across various dimensions: differences between normal and abnormal studies, generalization capabilities across healthcare sites, and error rates in identifying clinical findings. This benchmark reveals that automated chest X-ray report generation remains fundamentally unsolved, with significant performance gaps between normal and abnormal studies, where even top-performing models achieve less than 45% error-free reporting on abnormal cases, and substantial variability across healthcare institutions, indicating that robust, clinically-ready systems require continued development before widespread deployment.

Vol. 31
Microsoft (United States) (US), Harvard University (US), Seoul National University (KR), Johns Hopkins University (US), Duke University (US), University of Zurich (CH), Indian Institute of Technology Madras (IN), Gradient (United States) (US), Futures Group (United States) (US), Microsoft Research (India) (IN), Australian e-Health Research Centre (AU), CSIRO Health and Biosecurity (AU), Technology International Incorporated of Virginia (United States) (US), University of Glasgow (GB)
Openalex Percentile: Top 45%
COVID-19 diagnosis using AI
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.