Patient-level and duplicate-image data leakage inflate reported accuracy in OASIS and KAAD MRI benchmarks

Deep learning studies of Alzheimer’s disease classification often use two Kaggle-hosted MRI datasets: OASIS, and a widely used Kaggle Alzheimer’s MRI export sometimes falsely referred to as ADNI, here termed the Kaggle-Augmented Alzheimer’s Dataset (KAAD) to reflect its documented Kaggle provenance (Methods). Both are split into training and test sets at the image level without accounting for image provenance, creating an opportunity for identity leakage. In OASIS, patient identifiers recoverable from filenames show that 100% of test images share a patient with the training set under naive random splitting, across all 86,437 images. In KAAD, whose filenames are anonymized, exact-hash and validated perceptual-hash analyses estimate that roughly one-fifth of images are near-duplicates of another image in the set. Using three classifiers of increasing capacity, removing this leakage lowers mean test accuracy from 100.0% to about 46% for OASIS (p < = 0.00005) and from 68.0% to 58.9–62.8% for KAAD ( p = 0.0028-0.09); the OASIS collapse was essentially unchanged by a five-fold capacity increase, indicating it is not an artifact of an under-capacity baseline. A cross-dataset test across five seeds shows a classifier trained on OASIS performs at chance level on KAAD (25.0%) despite perfect training accuracy. These findings, consistent with prior reports, indicate that much published accuracy on these benchmarks reflects identity memorization rather than transferable diagnostic signal. Leakage-free, patient- and duplicate-aware split manifests are released for both datasets.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-14
DOI
https://doi.org/10.1038/s41598-026-77108-1
Primary Topic
Dementia and Cognitive Impairment Research
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Patient-level and duplicate-image data leakage inflate reported accuracy in OASIS and KAAD MRI benchmarks

V. Santhoshkumar, B. Lanitha
Scientific Reports
Dementia and Cognitive Impairment Research
article

Patient-level and duplicate-image data leakage inflate reported accuracy in OASIS and KAAD MRI benchmarks

V. Santhoshkumar, B. Lanitha
article en

Abstract

Deep learning studies of Alzheimer’s disease classification often use two Kaggle-hosted MRI datasets: OASIS, and a widely used Kaggle Alzheimer’s MRI export sometimes falsely referred to as ADNI, here termed the Kaggle-Augmented Alzheimer’s Dataset (KAAD) to reflect its documented Kaggle provenance (Methods). Both are split into training and test sets at the image level without accounting for image provenance, creating an opportunity for identity leakage. In OASIS, patient identifiers recoverable from filenames show that 100% of test images share a patient with the training set under naive random splitting, across all 86,437 images. In KAAD, whose filenames are anonymized, exact-hash and validated perceptual-hash analyses estimate that roughly one-fifth of images are near-duplicates of another image in the set. Using three classifiers of increasing capacity, removing this leakage lowers mean test accuracy from 100.0% to about 46% for OASIS (p < = 0.00005) and from 68.0% to 58.9–62.8% for KAAD ( p = 0.0028-0.09); the OASIS collapse was essentially unchanged by a five-fold capacity increase, indicating it is not an artifact of an under-capacity baseline. A cross-dataset test across five seeds shows a classifier trained on OASIS performs at chance level on KAAD (25.0%) despite perfect training accuracy. These findings, consistent with prior reports, indicate that much published accuracy on these benchmarks reflects identity memorization rather than transferable diagnostic signal. Leakage-free, patient- and duplicate-aware split manifests are released for both datasets.

Scientific Reports
Karpagam Academy of Higher Education (IN)
Dementia and Cognitive Impairment Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon