When AI Feedback Was Still in Its Infancy: An Exploratory Comparison of Early AI Feedback Attempts on Preservice Physics Teachers’ Reflective Writing

Reflective writing is a core component of teacher education, especially during practical internships. However, providing high-quality feedback on reflections is resource-intensive. This study examines descriptively observable associations between an early dual-feedback approach combining basic (automated) and elaborate (human-generated) feedback and structural features of preservice physics teachers’ reflective writing, prior to the widespread adoption of generative AI in education. Using an exploratory, non-equivalent, non-concurrent cohort design, we analyzed participant-level aggregates of written reflections from a non-intervention cohort (N = 22) and an intervention cohort (N = 32), applying a validated reflection-supporting model to assess structural composition and discursive elements of reflective writing. In the intervention, basic feedback was generated by a previously validated BERT-based machine learning model focusing on structural reflection elements, while elaborate feedback addressed content-related and pedagogical depth. In this study, the automated model was employed as an analytic measurement instrument drawing on validation work demonstrating its transferability across comparable reflection contexts. Quantitative analyses did not reveal systematic longitudinal growth in indicators of reflective writing quality in either cohort. Across comparable measurement points, descriptively different structural reflection profiles were observed between cohorts, without permitting causal or developmental interpretations. Feedback acceptance was high overall, although structural AI feedback was perceived as less personalized and less useful. These findings highlight the descriptive value of early, non-generative AI-based approaches for scalable structural diagnostics of reflective writing, while underscoring the continued importance of human-generated, content-focused feedback. The study establishes an empirical baseline for evaluating contemporary generative AI–based feedback systems in teacher education.