Limited transportability of a deep learning grading model for severe lumbar neural foraminal stenosis: external validation in two public cohorts
Background Aggregate accuracy and discrimination can remain acceptable even when an artificial intelligence system misses severe disease at its applied decision rule. We tested whether a locked lumbar magnetic resonance imaging (MRI) pipeline retained severe neural foraminal stenosis sensitivity and its development-defined safety operating point in an independent public cohort. Methods A two-stage ConvNeXt-Tiny pipeline was developed using LumbarDISC sagittal T2-weighted examinations. It localized eight level-by-side targets from L2–L3 to L5–S1 and graded each as normal/mild, moderate, or severe. Target-specific temperature scaling was applied. Using five-fold out-of-fold predictions, we selected the highest severe-probability threshold that achieved the investigator-selected sensitivity target of 0.90. All components were frozen before one-time evaluation in a 395-patient internal test set and the independent 465-patient AISSLab cohort. The primary endpoint was target-level severe-class sensitivity in AISSLab. Confidence intervals used 2,000 patient-cluster bootstrap samples. Results AISSLab contributed 2,190 labeled targets, including 205 severe events. Argmax severe-class sensitivity decreased from 0.404 (61/151; 95% CI, 0.315–0.487) internally to 0.185 (38/205; 95% CI, 0.135–0.242). The external-minus-internal difference was −0.219 (95% CI, −0.314 to −0.114). External severe-class area under the receiver operating characteristic curve was 0.885, indicating that preserved ranking did not ensure sensitivity at the frozen decision rule. The safety threshold achieved 0.824 sensitivity (169/205), below its 0.90 development target, with a positive predictive value of 0.288 and referral rate of 0.268. Dataset-provided boxes increased argmax sensitivity by only 0.029. Target-domain temperature scaling slightly improved calibration but did not change argmax sensitivity and increased referral workload at the fixed threshold. Conclusion This pipeline showed limited internal severe-class sensitivity and a further decrease externally. Independent validation should assess intended operating points while accounting for reference-standard differences and incomplete labels.
Authors
- 胡振顺
- 杨之源
- Huaguang Li
- Jingwei Chen
- Hua Liu
Institutions
- Tianjin University of Sport (CN)
- Hebei North University (CN)
Publication Details
- Journal
- Frontiers in Medicine
- Published
- 2026-11-17
- DOI
- https://doi.org/10.3389/fmed.2026.1965964
- Primary Topic
- Medical Imaging and Analysis
- Type
- article
- Field-Weighted Citation Impact
- 0.00