Exploring large language model’s capabilities in identifying science teacher PCK using lesson plans and open-ended questions

Pedagogical content knowledge (PCK) has been a cornerstone of science teacher education research, yet its practical application remains limited because of the non-standardized, time-intensive, and labor-intensive nature of PCK data collection and analysis. This study explores the potential of large language models (LLMs) to identify science teachers’ PCK levels on the topic of photosynthesis using open-ended responses and lesson plans. Iterative cycles of training and testing LLMs to assess various PCK components were conducted, introducing an innovative approach that utilized synthetic responses to train the models, which were subsequently validated with actual teacher responses. Findings indicate that synthetic data effectively trained LLMs to identify teacher PCK levels, though performance varied across PCK components. For instance, some models demonstrated strong performance in assessing Knowledge of Instructional Strategies and Representations, as well as Knowledge of Assessment of Science Learning, but struggled with Knowledge of Student Understanding. The study also examined the relationships between teacher characteristics (e.g. self-efficacy, years of experience, and National Board Certification) and PCK levels identified by both humans and LLMs. Results showed some alignment in correlations for particular PCK components, though consistency varied across models. Furthermore, the human-machine reliability for identifying PCK levels from lesson plans approached human-human reliability, with some values exceeding 0.80. These findings highlight the significant potential of LLMs have to advance and scale science teacher PCK research by incorporating multiple data sources. Challenges and opportunities associated with identifying PCK levels using LLMs are discussed, providing implications for future research and science teacher education.