Hey AI, Are You Sure? Analyzing Reflective Prompting Attempts in AI-Based Writing Assessment

As artificial intelligence (AI) becomes increasingly integrated into education, its role in the assessment of writing skills has emerged as a significant area of research. This study investigates the reliability and consistency of a custom-built, GPT-based model specifically designed to evaluate B1-level English opinion paragraphs. A dataset consisting of 40 texts (20 written by students and 20 generated by AI) was evaluated across six separate sessions. In the first three sessions, the model assigned scores based on a standard rubric. In the remaining three sessions, it was prompted to re-evaluate its own scores based on reflective questions that suggested it might have “overestimated,” “underestimated,” or “overestimated or underestimated” the original score; thus introducing positive, negative, and neutral reflective prompting. Guided by six research questions, the study examines the internal consistency of the model and its responsiveness to such reflective prompting. Intraclass correlation coefficients (ICCs) indicated excellent reliability across all conditions (ICC > .94). Predictable changes were observed in scoring behavior depending on the prompt direction, particularly in rubric components involving higher-order cognitive skills (i.e., content and organization), whereas grammar and vocabulary scores remained stable. Although the limited sample size constrains the generalizability of the findings, these findings demonstrate that AI-based scoring is not only reliable but also adaptable to metacognitive prompts, offering valuable insights for scalable, rubric-aligned assessment models. The study highlights that GPT-based tools can serve not only as dependable evaluators of student writing but also as instruments that promote self-assessment, support rater training, and foster more equitable feedback in educational settings.