Behavioral Assessment Reliability in Clinical Phenotyping and Biomarker Research for Autism

Autism Spectrum Disorder standardized behavioral assessments provide quantitative measures of symptoms, yet their reliability and consistency have not been systematically evaluated. We present the first large-scale comparative analysis of four widely used assessments. We analyzed behavioral assessments across three autism cohorts using correlations, clustering, and diagnostic agreement analyses. We related behavioral variation to genetic and imaging data to evaluate biomarker associations. Sentence-level embeddings generated by large language models reveal substantial semantic overlap across instruments. Nonetheless, behavioral scores are weakly correlated (0.26 ± 0.21), and diagnostic classification shows only 65-80% agreement between tests. These patterns hold across three datasets comprising N = 1 954. None of the assessments show consistent associations with widely studied MRI or genetic biomarkers. These findings expose critical inconsistencies among widely used autism assessments and underscore the need for more reliable tools to support precision phenotyping, biomarker discovery, and individualized care. Rather than diminishing the utility of behavioral assessment in autism, the inconsistencies identified here highlight a critical opportunity to refine how behavioral phenotypes are defined and operationalized.