Beyond 99%: Auditing Leakage, Shortcuts and Calibration in COVID-19 CT Classification

A 99.76%-accuracy COVID-19 CT classifier turns out to measure much less than it claims. I audited it end to end and wrote it up as a preprint (v1.0, October 12 2026). Key findings : • Showed that slice-level splits inflate accuracy by placing near-duplicate slices in both train and test: recovered pseudo-paGent groups by clustering deep and thumbnail features, and measured accuracy falling from 99.42% to 95.39% undergrouped splits (error up to 8.0x).• IdenGfied shortcut learning: a metadata-only model reached 0.996 balanced accuracy on random splits (0.630 undergrouped splits), and the classifier kept ~0.93 balanced accuracy with the lungs masked out, confirmed with Grad-CAMlung-enrichment analysis.• Tested external validity and calibraGon on Sao Paulo and Tehran (COVID-CT-MD) data: AUROC ≈0.83 externally, nearchance (0.43-0.49) for COVID vs community-acquired pneumonia, and calibraGon error rising from 0.02 to 0.19-0.53;released a one-script reproducible pipeline and a 7-step audit protocol.