Patient-level and duplicate-image data leakage inflate reported accuracy in OASIS and KAAD MRI benchmarks
Deep learning studies of Alzheimer’s disease classification often use two Kaggle-hosted MRI datasets: OASIS, and a widely used Kaggle Alzheimer’s MRI export sometimes falsely referred to as ADNI, here termed the Kaggle-Augmented Alzheimer’s Dataset (KAAD) to reflect its documented Kaggle provenance (Methods). Both are split into training and test sets at the image level without accounting for image provenance, creating an opportunity for identity leakage. In OASIS, patient identifiers recoverable from filenames show that 100% of test images share a patient with the training set under naive random splitting, across all 86,437 images. In KAAD, whose filenames are anonymized, exact-hash and validated perceptual-hash analyses estimate that roughly one-fifth of images are near-duplicates of another image in the set. Using three classifiers of increasing capacity, removing this leakage lowers mean test accuracy from 100.0% to about 46% for OASIS (p < = 0.00005) and from 68.0% to 58.9–62.8% for KAAD ( p = 0.0028-0.09); the OASIS collapse was essentially unchanged by a five-fold capacity increase, indicating it is not an artifact of an under-capacity baseline. A cross-dataset test across five seeds shows a classifier trained on OASIS performs at chance level on KAAD (25.0%) despite perfect training accuracy. These findings, consistent with prior reports, indicate that much published accuracy on these benchmarks reflects identity memorization rather than transferable diagnostic signal. Leakage-free, patient- and duplicate-aware split manifests are released for both datasets.
Authors
- V. Santhoshkumar
- B. Lanitha
Institutions
- Karpagam Academy of Higher Education (IN)
Publication Details
- Journal
- Scientific Reports
- Published
- 2026-11-14
- DOI
- https://doi.org/10.1038/s41598-026-77108-1
- Primary Topic
- Dementia and Cognitive Impairment Research
- Type
- article
- Field-Weighted Citation Impact
- 0.00