Evaluating the Potential of LLMs for Better Short Answer Scoring

Automated Short Answer Grading (ASAG) has emerged as a promising tool for the challenge of assessing open student responses in an efficient and scalable manner as manual grading of such open short answers is labor-intensive and time-consuming. In this study, we present several ways of refining LLMs to fit the task of grading student short-answer responses robustly, fairly, and consistently, including a task-specific approach and a combined variant, being able to assess different tasks within the same model. In this regard, we explore two key questions: (1) Are transformer-based models suitable for short-answer grading? (2) Can a single transformer-based model effectively generalize across diverse tasks? The experimental results showed the significant potential of fine-tuned LLMs in ASAG. We further compared different fine-tuning strategies and the experimental results showed that full-fine-tuned models outperformed other fine-tuning approaches.