A Reproducible Benchmark for Leakage-Resistant Machine Learning on Multi-Site Brain Connectivity Graphs

Background/Objectives: Machine-learning studies of autism spectrum disorder from resting-state functional magnetic resonance imaging routinely report near-perfect accuracy, yet the models rarely transfer to an unseen scanner. We audited a pipeline of the kind this literature builds, which reported an accuracy of 0.999, and traced that number to five data-leakage pathways. Methods: Under a nested leave-one-site-out protocol on 871 participants from the Autism Brain Imaging Data Exchange, in which the held-out site influenced nothing but model selection, the threshold, the combiner, or the site correction, we closed every pathway, priced each by reopening it, and validated on a locked external cohort. Results: With all five pathways closed, the graph attention network reached an area under the curve of 0.648 and did not outperform a support vector machine on the same connectivity (0.679; p = 0.009). Decision-level fusion added nothing detectable (0.682, p = 0.72). A network identified its training participants better than it detected autism (0.710 versus 0.610); both questionnaire benchmarks proved label-tautological, and performance held on the external cohort. Conclusions: Reported gains in this literature should be read as claims about evaluation protocols until shown otherwise on an untouched acquisition site; all predictions, models, and code are released.