A new Q‐matrix validation method based on signal detection theory

The Q-matrix is a crucial component of cognitive diagnostic theory and an important basis for the research and practical application of cognitive diagnosis. In practice, the Q-matrix is typically developed by domain experts and may contain some misspecifications, so it needs to be refined using Q-matrix validation methods. Based on signal detection theory, this paper puts forward a new Q-matrix validation method (i.e., <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:semantics><mml:mrow><mml:mi>β</mml:mi></mml:mrow> </mml:semantics> </mml:math> method) and then conducts a simulation study to compare the new method with existing methods. The results show that when the model is DINA (deterministic inputs, noisy 'and' gate), the <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:semantics><mml:mrow><mml:mi>β</mml:mi></mml:mrow> </mml:semantics> </mml:math> method outperforms the existing methods under all conditions; under the generalized DINA (G-DINA) model, the method still has the highest validation rate when the sample size is small, and the item quality is high or the rate of Q-matrix misspecification is ≥.4. Finally, a sub-dataset of the PISA 2000 reading assessment is analysed to evaluate the reliability of the <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:semantics><mml:mrow><mml:mi>β</mml:mi></mml:mrow> </mml:semantics> </mml:math> method.