Constructing a Computerized Adaptive Test for Multidimensional Mathematical Competence: Development, Simulation, and Validation

This study utilized Item Response Theory (IRT) and Computerized Adaptive Testing (CAT) techniques to develop and preliminarily evaluate a multidimensional Mathematics Competence Assessment and Diagnosis (MCAD). The research consisted of three studies: First, we followed a five-step process to construct the item bank, which included Item Construction, Booklet Design, Participants, Procedure, and Data Analysis. Second, we conducted a simulation study to assess measurement precision and testing efficiency. The results indicated that MCAD’s measurement precision exceeded that of unidimensional CAT and random administration of items, requiring less than 33% of the items needed by these methods to achieve equivalent reliability. Third, after developing the MCAD system, we carried out a preliminary empirical study involving 105 tenth-grade students. We used their Comprehensive Assessment Program for Junior High School Students (CAP) mathematics test as a criterion to examine the MCAD’s criterion-related evidence, which was found to be over 0.7. However, the high correlations among dimensions and the characteristics of the empirical sample suggest that dimension-specific interpretations should be made cautiously. Overall, the findings provide preliminary support for MCAD as a potentially efficient tool for assessing mathematical competence.