Lessons Learned in Assessing Student Reflections with LLMs

Advances in Large Language Models (LLMs) have sparked growing interest in their potential as explainable text evaluators.While LLMs have shown promise in assessing machinegenerated texts in tasks such as summarization and machine translation, their effectiveness in evaluating human-written content-such as student writing in classroom settings-remains underexplored.In this paper, we investigate LLM-based specificity assessment of student reflections written in response to prompts, using three instruction-tuned models.Our findings indicate that although LLMs may underperform compared to simpler supervised baselines in terms of scoring accuracy, they offer a valuable interpretability advantage.Specifically, LLMs can generate explanations that are faithful, non-repetitive, and exhibit high fidelity with their input, suggesting potential for enhancing the transparency and usability of automated specificity scoring systems.