Parametric Bootstrap Mantel‐Haenszel Statistic for Aggregated Testlet Effects

Abstract While testlets have proven useful for assessing complex skills, the stem shared by multiple items often induces correlations between responses, leading to violations of local independence (LI), which can result in biased parameter and ability estimates. Diagnostic procedures for detecting testlet effects typically involve model comparisons testing for the inclusion of extra testlet parameters or, at the item level, testing for pairwise LI. Rosenbaum's adaptation of the Mantel‐Haenszel (MH) ‐statistic belongs to the latter category. The MH ‐statistic has also been used in cognitive diagnosis for detecting violations of LI and for the identification of testlet effects. However, this approach is not without limitations, as it lacks a rationale for integrating multiple pairwise MH ‐statistics and any notion of the sampling distribution of such an integrated statistic. In this article, a procedure for integrating multiple pairwise MH ‐statistics to evaluate testlet effects in cognitive diagnosis is proposed. The unknown sampling distribution issue is addressed by implementing a parametric bootstrap resampling scheme. Results from simulation studies demonstrate the performance of the proposed parametric bootstrap testlet MH ‐statistic, and its application to the 2015 PISA Collaborative Problem Solving (CPS) data set illustrates the method's practical merits.