Validating AI-assisted evaluation of open science practices in brain sciences: ChatGPT, Claude and human expert comparisons

Abstract This study investigates the efficacy of AI-assisted evaluation of open science practices in brain sciences, comparing ChatGPT 4 and Claude 3.5 Sonnet against human expert assessment. We analysed 100 randomly selected journal articles across various brain science disciplines using a six-item transparency checklist. Three human experts and two AI chatbots independently evaluated the articles. Intraclass correlation analyses revealed moderate-to-good agreement between human consensus scores and AI evaluations, with pairwise correlations providing convergent evidence of substantial correspondence. Both chatbots demonstrated high concordance with humans in assessing code sharing, materials availability, preregistration and sample size rationales. However, they struggled with accurately identifying the presence of data availability statements and assessing public accessibility of shared data. Importantly, AI assessments were restricted to information stated within the article; external links or repository contents were not cross-checked owing to limited web search capability of AI chatbots. These findings suggest that AI chatbots can effectively support the evaluation of some open science practices and potentially expedite the assessment process in academic research. However, their limitations in certain areas highlight the continued importance of human oversight in ensuring comprehensive and accurate evaluations of scientific transparency.