Benchmarking of AI and Radiologists for Indeterminate Lung Nodule Malignancy Risk Estimation on Screening CT: The LUNA25 Challenge

Purpose To compare the performance of an artificial intelligence (AI) system with that of radiologists for estimating malignancy risk of indeterminate-size nodules (5–15 mm) on low-dose CT (LDCT) within a standardized and transparent evaluation framework. Materials and Methods Teams participating in the AI study had access to a public dataset of 555 malignant and 5608 benign nodules on 4069 baseline LDCT scans from the National Lung Screening Trial (NLST) to develop AI systems. External testing was performed on 156 malignant and 312 benign size-matched nodules, all of indeterminate size, from 463 baseline scans collected from three large European lung cancer screening trials between 2004 and 2018, and the best-performing AI system (based on area under the receiver operating characteristic curve [AUC]) was selected. An observer study was conducted in which radiologists assessed 300 randomly selected nodules (100 malignant, 200 benign) from the external test set. Radiologists categorized nodules as low, intermediate, or high-risk, and the ≥ intermediate-risk threshold (intermediate or high-risk) was used to define a positive test. The selected AI system was compared with radiologists on this subset using the AUC. Results The selected AI system demonstrated superior performance to the 65 radiologists’ average (AUC, 0.78 (95% CI: 0.73, 0.84) vs 0.70 (95% CI: 0.65, 0.74), P = .001). When using the ≥ intermediate-risk threshold, the AI system correctly classified 12% more malignant nodules at matched specificity and yielded 20% fewer false positives at matched sensitivity. Conclusion The selected AI system was superior to radiologists in estimating malignancy risk of indeterminate lung nodules on LDCT. ©RSNA, 2026

Authors

Institutions

Publication Details

Journal
Radiology Artificial Intelligence
Published
2026-06-24
DOI
https://doi.org/10.1148/ryai.260179
Primary Topic
Lung Cancer Diagnosis and Treatment
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Benchmarking of AI and Radiologists for Indeterminate Lung Nodule Malignancy Risk Estimation on Screening CT: The LUNA25 Challenge

Stephen Lam, F Gleeson, Rozemarijn Vliegenthart, Colin Jacobs et al.
Radiology Artificial Intelligence
Lung Cancer Diagnosis and Treatment
article

Benchmarking of AI and Radiologists for Indeterminate Lung Nodule Malignancy Risk Estimation on Screening CT: The LUNA25 Challenge

Stephen Lam, F Gleeson, Rozemarijn Vliegenthart, Colin Jacobs, Firdaus A. A. Mohamed Hoesein, Zaigham Saghir, Erik H.F.M. van der Heijden, Cornelia Schaefer‐Prokop, Ugo Pastorino, Mathias Prokop, Geert Litjens, Marjolein A. Heuvelmans, Geertruida H. de Bock, Ernst T. Scholten, Dré Peeters, Annemiek Snoeckx, Noa J.C. Antonissen, Mario Silva, Hester Gietema, Bogdan Obreja
article en

Abstract

Purpose To compare the performance of an artificial intelligence (AI) system with that of radiologists for estimating malignancy risk of indeterminate-size nodules (5–15 mm) on low-dose CT (LDCT) within a standardized and transparent evaluation framework. Materials and Methods Teams participating in the AI study had access to a public dataset of 555 malignant and 5608 benign nodules on 4069 baseline LDCT scans from the National Lung Screening Trial (NLST) to develop AI systems. External testing was performed on 156 malignant and 312 benign size-matched nodules, all of indeterminate size, from 463 baseline scans collected from three large European lung cancer screening trials between 2004 and 2018, and the best-performing AI system (based on area under the receiver operating characteristic curve [AUC]) was selected. An observer study was conducted in which radiologists assessed 300 randomly selected nodules (100 malignant, 200 benign) from the external test set. Radiologists categorized nodules as low, intermediate, or high-risk, and the ≥ intermediate-risk threshold (intermediate or high-risk) was used to define a positive test. The selected AI system was compared with radiologists on this subset using the AUC. Results The selected AI system demonstrated superior performance to the 65 radiologists’ average (AUC, 0.78 (95% CI: 0.73, 0.84) vs 0.70 (95% CI: 0.65, 0.74), P = .001). When using the ≥ intermediate-risk threshold, the AI system correctly classified 12% more malignant nodules at matched specificity and yielded 20% fewer false positives at matched sensitivity. Conclusion The selected AI system was superior to radiologists in estimating malignancy risk of indeterminate lung nodules on LDCT. ©RSNA, 2026

Radiology Artificial Intelligence
University of Copenhagen (DK), University of Parma (IT), BC Cancer Agency (CA), University Medical Center Groningen (NL), University of British Columbia (CA), Radboud University Nijmegen (NL), University of Antwerp (BE), Utrecht University (NL), Maastricht University Medical Centre (NL), Gentofte Hospital (DK), Radboud University Medical Center (NL), University Medical Center Utrecht (NL), Maastricht University (NL), University of Oxford (GB), Antwerp University Hospital (BE), Radboud Institute for Molecular Life Sciences (NL), Oncode Institute (NL), Science Oxford (GB), Amsterdam University Medical Centers (NL), Fondazione IRCCS Istituto Nazionale dei Tumori (IT), Meander Medisch Centrum (NL), Maastro Clinic (NL), IT University of Copenhagen (DK), University of Amsterdam (NL)
Good health and well-being
Openalex Percentile: Top 7%
Lung Cancer Diagnosis and Treatment
DOISourceOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.