Exploring the Pedagogical Potential of Generative AI in English Literature Education: Comparative Assessment of Parody Assignments Through Gemini Pro, GPT-3.5, and GPT-4

This study investigates how effectively generative AI can provide feedback on creative writing for English literature education. In order to examine the reliability and validity of the automated feedback, 40 movie parody assignments written by 40 English major undergraduates were evaluated by three AI models (Gemini 1.0 Pro, GPT-3.5-turbo, and GPT-4-turbo) with four constructs. Statistical analysis of the three models’ scores, generated at four temperature settings (0, 0.4, 0.7, 1.0) with 30 iterations per setting, are as follows: First, all three models showed high reliability in scoring on a 0 to 5 scale. Second, rank-based scoring analysis revealed differences in reliability among the models. Gemini 1.0 Pro demonstrated high reliability only at 0 temperature, whereas GPT-4 maintained consistent reliablity across all temperature settings. Third, scoring validity varied significantly among models. Validity assessments through the review of student assignments and the analysis of rank-based scoring contrasted the limited validity of Gemini 1.0 Pro with the high validity of GPT-4’s consistent reliability scoring. However, GPT-4’s tendency toward holistic scoring should be considered. These findings suggest that all three models, which can consistently provide general-level feedback, would be limited in providing optimized feedback. The significance of this study lies in its empirical validation of generative AI’s potential as a feedback provider in creative writing education.