When Near-Perfect Accuracy Is a Warning Sign: Frame-Level Data Leakage in 3D CNN Classification of the Alzheimer's Disease Spectrum from fMRI Volumes

Abstract Deep learning studies on neuroimaging data frequently report accuracies close to 100%, yet such results can arise from data leakage rather than from learned disease signal. This paper is a case study of that failure in the author’s own work. A 3D convolutional neural network (CNN) was trained to separate Alzheimer’s disease (AD), mild cognitive impairment (MCI), early MCI (EMCI) and cognitively normal (CN) participants using 191,428 individual 3D volumes extracted from functional MRI (fMRI) acquisitions in the Alzheimer’s Disease Neuroimaging Initiative (ADNI). Each time frame was treated as an independent image, and the pooled frames were split at random, at the image level, into training and validation sets. The network reached 100% validation accuracy by epoch 14 and made no errors on any of the 191,428 volumes. We show that this outcome is explained by frame-level leakage: frames of the same participant are nearly identical, so a model can recognise individuals instead of disease stages, and a random split places almost every participant on both sides. We document the training signatures that exposed the problem, relate them to leakage patterns reported in the Alzheimer’s literature, quantify why image-level splitting of fMRI frames guarantees subject overlap, and propose a subject-level evaluation protocol. The corrected experiment is left to future work; the contribution of this paper is a transparent, reproducible account of how a plausible pipeline yields an invalid result, and how to detect it.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-16
DOI
https://doi.org/10.21203/rs.3.rs-11694221/v1
Primary Topic
Dementia and Cognitive Impairment Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

When Near-Perfect Accuracy Is a Warning Sign: Frame-Level Data Leakage in 3D CNN Classification of the Alzheimer's Disease Spectrum from fMRI Volumes

ARIFUL Islam
1 links
Research Square
Dementia and Cognitive Impairment Research
preprint

When Near-Perfect Accuracy Is a Warning Sign: Frame-Level Data Leakage in 3D CNN Classification of the Alzheimer's Disease Spectrum from fMRI Volumes

ARIFUL Islam
preprint en

Abstract

Abstract Deep learning studies on neuroimaging data frequently report accuracies close to 100%, yet such results can arise from data leakage rather than from learned disease signal. This paper is a case study of that failure in the author’s own work. A 3D convolutional neural network (CNN) was trained to separate Alzheimer’s disease (AD), mild cognitive impairment (MCI), early MCI (EMCI) and cognitively normal (CN) participants using 191,428 individual 3D volumes extracted from functional MRI (fMRI) acquisitions in the Alzheimer’s Disease Neuroimaging Initiative (ADNI). Each time frame was treated as an independent image, and the pooled frames were split at random, at the image level, into training and validation sets. The network reached 100% validation accuracy by epoch 14 and made no errors on any of the 191,428 volumes. We show that this outcome is explained by frame-level leakage: frames of the same participant are nearly identical, so a model can recognise individuals instead of disease stages, and a random split places almost every participant on both sides. We document the training signatures that exposed the problem, relate them to leakage patterns reported in the Alzheimer’s literature, quantify why image-level splitting of fMRI frames guarantees subject overlap, and propose a subject-level evaluation protocol. The corrected experiment is left to future work; the contribution of this paper is a transparent, reproducible account of how a plausible pipeline yields an invalid result, and how to detect it.

Research Square
East Delta University (BD)
Good health and well-being
Dementia and Cognitive Impairment Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon