Considering Multiple Sources of Validity Evidence Can Help to Address Challenges in the Development of Pedagogical Content Knowledge (PCK) Multiple-Choice Items

Abstract The use of multiple-choice (MC) instruments to assess pedagogical content knowledge (PCK) has advantages in terms of test economy and objectivity, but it also poses challenges, for example, in terms of adequately capturing the intended construct. To help address these challenges, we developed and evaluated a new instrument to assess science teachers’ PCK of scientific reasoning in biology contexts (PCK SR -bio), considering multiple sources of validity evidence. First, 12 MC items were developed to assess crucial PCK components for three scientific reasoning skills. Subsequently, the correlation of corresponding content knowledge (CK) with the PCK SR -bio score was tested with 67 master’s students. In addition, the instrument was used in a cross-sectional study with 165 students ( n = 29 bachelor, n = 115 master, n = 21 school-based preservice teachers), and the internal consistency as well as the correlation of the test score with the educational level was determined. An analysis of the response processes of 10 bachelor’s students showed that they more often referred to PCK when selecting an attractor and more often (rather intuitively) to other knowledge when selecting a distractor. In the cross-sectional study, the internal consistency was relatively low but increased with higher educational level. A correlation was found between the test score and CK but not between the test score and the educational level. Our results show that considering multiple sources of validity evidence can help to address common challenges in developing MC–PCK instruments. The results and limitations are discussed, and recommendations are made for the development of MC instruments to assess PCK in general.