Limited transportability of a deep learning grading model for severe lumbar neural foraminal stenosis: external validation in two public cohorts

Background Aggregate accuracy and discrimination can remain acceptable even when an artificial intelligence system misses severe disease at its applied decision rule. We tested whether a locked lumbar magnetic resonance imaging (MRI) pipeline retained severe neural foraminal stenosis sensitivity and its development-defined safety operating point in an independent public cohort. Methods A two-stage ConvNeXt-Tiny pipeline was developed using LumbarDISC sagittal T2-weighted examinations. It localized eight level-by-side targets from L2–L3 to L5–S1 and graded each as normal/mild, moderate, or severe. Target-specific temperature scaling was applied. Using five-fold out-of-fold predictions, we selected the highest severe-probability threshold that achieved the investigator-selected sensitivity target of 0.90. All components were frozen before one-time evaluation in a 395-patient internal test set and the independent 465-patient AISSLab cohort. The primary endpoint was target-level severe-class sensitivity in AISSLab. Confidence intervals used 2,000 patient-cluster bootstrap samples. Results AISSLab contributed 2,190 labeled targets, including 205 severe events. Argmax severe-class sensitivity decreased from 0.404 (61/151; 95% CI, 0.315–0.487) internally to 0.185 (38/205; 95% CI, 0.135–0.242). The external-minus-internal difference was −0.219 (95% CI, −0.314 to −0.114). External severe-class area under the receiver operating characteristic curve was 0.885, indicating that preserved ranking did not ensure sensitivity at the frozen decision rule. The safety threshold achieved 0.824 sensitivity (169/205), below its 0.90 development target, with a positive predictive value of 0.288 and referral rate of 0.268. Dataset-provided boxes increased argmax sensitivity by only 0.029. Target-domain temperature scaling slightly improved calibration but did not change argmax sensitivity and increased referral workload at the fixed threshold. Conclusion This pipeline showed limited internal severe-class sensitivity and a further decrease externally. Independent validation should assess intended operating points while accounting for reference-standard differences and incomplete labels.

Authors

Institutions

Publication Details

Journal
Frontiers in Medicine
Published
2026-11-17
DOI
https://doi.org/10.3389/fmed.2026.1965964
Primary Topic
Medical Imaging and Analysis
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Limited transportability of a deep learning grading model for severe lumbar neural foraminal stenosis: external validation in two public cohorts

胡振顺, 杨之源, Huaguang Li, Jingwei Chen et al.
6 links
Frontiers in Medicine
Medical Imaging and Analysis
article

Limited transportability of a deep learning grading model for severe lumbar neural foraminal stenosis: external validation in two public cohorts

胡振顺, 杨之源, Huaguang Li, Jingwei Chen, Hua Liu
article en

Abstract

Background Aggregate accuracy and discrimination can remain acceptable even when an artificial intelligence system misses severe disease at its applied decision rule. We tested whether a locked lumbar magnetic resonance imaging (MRI) pipeline retained severe neural foraminal stenosis sensitivity and its development-defined safety operating point in an independent public cohort. Methods A two-stage ConvNeXt-Tiny pipeline was developed using LumbarDISC sagittal T2-weighted examinations. It localized eight level-by-side targets from L2–L3 to L5–S1 and graded each as normal/mild, moderate, or severe. Target-specific temperature scaling was applied. Using five-fold out-of-fold predictions, we selected the highest severe-probability threshold that achieved the investigator-selected sensitivity target of 0.90. All components were frozen before one-time evaluation in a 395-patient internal test set and the independent 465-patient AISSLab cohort. The primary endpoint was target-level severe-class sensitivity in AISSLab. Confidence intervals used 2,000 patient-cluster bootstrap samples. Results AISSLab contributed 2,190 labeled targets, including 205 severe events. Argmax severe-class sensitivity decreased from 0.404 (61/151; 95% CI, 0.315–0.487) internally to 0.185 (38/205; 95% CI, 0.135–0.242). The external-minus-internal difference was −0.219 (95% CI, −0.314 to −0.114). External severe-class area under the receiver operating characteristic curve was 0.885, indicating that preserved ranking did not ensure sensitivity at the frozen decision rule. The safety threshold achieved 0.824 sensitivity (169/205), below its 0.90 development target, with a positive predictive value of 0.288 and referral rate of 0.268. Dataset-provided boxes increased argmax sensitivity by only 0.029. Target-domain temperature scaling slightly improved calibration but did not change argmax sensitivity and increased referral workload at the fixed threshold. Conclusion This pipeline showed limited internal severe-class sensitivity and a further decrease externally. Independent validation should assess intended operating points while accounting for reference-standard differences and incomplete labels.

Frontiers in MedicineVol. 13
Tianjin University of Sport (CN), Hebei North University (CN)
Good health and well-being
Medical Imaging and Analysis
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon