Modeling Item Responses to Understand Change over Time and Diagnostic Needs in Tandem

Abstract Achievement tests are often designed to measure student growth over time using vertical scales while also providing diagnostic information on mastery of specific skill areas. However, traditional assessments rarely integrate psychometric models appropriate for both purposes. This study examines an approach combining vertical scaling with Diagnostic Classification Models (DCMs) to provide both overall achievement scores and subtest‐level diagnostic classifications to provide growth and diagnostic information simultaneously. We compared four model specifications against traditional unidimensional vertical scaling using empirical data from the Virginia Language and Literacy Screening System with 17,302 kindergarten and 18,339 first‐grade students. We evaluated model fit, score comparability, and classification reliability, and added Monte Carlo simulations to assess parameter recovery. Results revealed important tradeoffs between approaches. While no single approach provided an ideal solution, second‐order DCMs with appropriate constraints offer practical viability for operational assessment contexts that wish to measure student growth while also providing diagnostic information.