Exploring Query-Driven Centroiding and Embedding Strategies for Automated Thematic Analysis in Physics Education

Understanding how Physics Education Research (PER) has changed and matured can help researchers and materially improve physics teaching and learning.The advent of large language models (LLMs) and embedding-based NLP techniques opens new avenues for this work by enabling the analysis of large corpora with minimal manual coding.This preliminary study builds off the work of [T.O. B. Odden et al., Physical Review Physics Education Research 20, 020151 (2024)].We explore the use of text embeddings and retrieval-augmented generation (RAG)-like methods without generative AI to analyze themes across a random sample of 94 articles from major PER journals.Specifically, we examine how two methodological choices affect topic modeling performance: (1) representing articles using a single embedding versus multiple sentence-level embeddings, and (2) deriving topic centroids from representative texts versus researcher-defined queries.All results are evaluated against a human-coded dataset focused on four overarching thematic categories: teacher-centered, student-centered, physics content, and journal business.Our findings inform best practices for researchers seeking scalable, interpretable, and low-barrier approaches to literature analysis in PER.