Questioning the Benchmarks: Hidden Circularity in AQ-10 Autism Screening Datasets

Autism spectrum disorder (ASD) screening increasingly relies on machine learning models trained on the AQ-10 and Q-CHAT questionnaire datasets, and these models are widely reported to reach 95 to 99 percent accuracy, so confirming whether that accuracy reflects genuine diagnostic learning is vital. This paper examines whether that accuracy reflects genuine diagnostic learning or is simply an artifact of how the datasets are built. The study compares six model families, namely Logistic Regression, Random Forest, Support Vector Machine, XGBoost, FT-Transformer, and TabNet, across three UCI AQ-10 age variants: Child (n=290), Adolescent (n=103), and Adult (n=699). Most emphasis is put on separating the behavioral items that define the diagnostic label from the demographic covariates that do not, since the label itself is a threshold applied to those same items. The effectiveness of this separation is measured using a normalized Circularity Index alongside item-only and demographics-only accuracy. Evaluation of the results has shown that item-based accuracy stays at 93.2 to 100 percent even after the pre-computed result score column is removed, while demographics-only accuracy falls to 50.7 to 71.5 percent, at or below the majority-class baseline in every dataset, giving a mean Circularity Index of 0.947. The same pattern is confirmed on two independent domains outside autism screening: DASS-42 depression screening (n=35,481, CI 0.984) and the Rosenberg Self-Esteem Scale (n=39,076, CI 0.896). Lastly, a diagnostic heuristic named AutoCirc is introduced to flag this kind of circularity automatically using mutual information, reproducing the manual Circularity Index on three of five datasets, and a constructive eight-of-ten-item AQ-10 short form is proposed as a practical outcome of the same analysis. The results of the study confirm that high reported accuracy on these datasets reflects the scoring arithmetic of the screening instrument rather than a learned diagnostic pattern, though this does not call the AQ-10 into question as a clinical screening tool when used as intended. Further work shall seek to extend this audit to additional screening instruments, such as PHQ-9 and GAD-7, and to datasets carrying independent clinical diagnosis labels.