Assessing a combined human coding and natural language processing method for qualitative analysis in physics education research

As artificial intelligence becomes increasingly integrated into education research, it is critical to assess the capabilities and limitations of such tools in complex disciplinary contexts.This study explores the effectiveness of natural language processing (NLP) for topic identification using the BERTopic package to analyze undergraduate quantum mechanics syllabi.By comparing AI-generated topics with human-coded themes from a corpus of syllabi from 50 US institutions, we find that while NLP reliably identifies broad and structural elements of syllabi (e.g., policies, textbook references), it is limited in its ability to capture the nuanced and context-dependent content typical of upper-level physics instruction.The AI-derived topics were often underrepresented and diverged from expert human interpretation, particularly in areas requiring deep disciplinary knowledge.Our findings support the use of NLP as a complementary tool for theme generation and cross-validation, but emphasize the continued necessity of expert human analysis for rigorous and meaningful educational research.