Preliminary Psychometric Analysis of an Online Diagnostic Assessment of Statistical Reasoning for Psychology Students: A Cognitive Psychology Perspective

Digital tools make diagnostic tests easy to build, but convenience does not guarantee that scores reflect the intended thinking processes. This study examined item quality and reliability of Draft 1 of the Statistical Reasoning Diagnostic Assessment for Psychology Students (SRDA-PSY), an online two-tier diagnostic test covering six domains of statistical reasoning, interpreting the results through cognitive psychology principles. Forty-one psychology students took the test via Google Forms. Classical test theory analyses (difficulty, discrimination, distractor effectiveness, two-tier response patterns, and KR-20) were triangulated with a single expert review. The mean score was 83%, and 26 of 41 items were very easy. All 15 items of moderate or easy difficulty discriminated well, but only 21% of distractors were functional. The key was the longest option in 57 of 63 option sets, and the six items without this cue were much harder (mean p = .53 vs. .89), suggesting that many items could be answered with an option-form heuristic rather than analytic reasoning. On two-tier items, students often endorsed a correct general principle but failed to apply it to the case, consistent with the view that intuitive and scientific conceptions can coexist. KR-20 was .90 but dropped to .78 when the two lowest-scoring respondents were removed, and expert review and empirical data detected different weaknesses. The content of SRDA-PSY is promising, but option form must be revised so that scores reflect statistical reasoning. Digital assessment development, including AI-assisted item writing, should combine expert review, empirical tryouts, and automated audits of option-form cues.