Calibration and Design Considerations for Multidimensional Multistage Testing

Abstract Multidimensional multistage testing (MMST) combines multidimensional item response theory (MIRT) with adaptive multistage testing design to measure multiple related constructs efficiently. Despite its potential for large‐scale assessments, little research has examined calibration methods under MMST. This study compared concurrent calibration, fixed item parameter calibration, and concurrent calibration with multiple panels using a simulated three‐stage MMST design with a within‐item two‐dimensional MIRT structure. Sample size, trait correlation, and routing stage length were manipulated across 36 conditions. Results showed that larger sample sizes and longer routing stages generally reduced RMSE for discrimination parameters, whereas higher trait correlations increased discrimination RMSE. In contrast, ability RMSE decreased as trait correlation increased, while the effects of sample size and routing stage length were less uniform. Difficulty RMSE remained relatively stable across conditions. Differences among calibration methods were generally small, with no significant pairwise differences among methods.