Enhancing Automated Short Answer Grading Using Large Language Models Through In-Context Learning, Chain-of-Thought Prompting, and Stringent Coefficients

This paper aims to optimize large language models for scoring short-answer questions in datasets. The study analyzed 4,500 short-answer responses sourced from an e-learning platform at a university, encompassing five academic disciplines, and assessed nine leading large language models. During the evaluation process, it was observed that the base models exhibited hallucination phenomena, which compromised score consistency. To address these issues, the researchers employed in-context learning and chain-of-thought prompting techniques as optimization strategies. The results indicated that these methods significantly improved model performance. Notably, while the chain-of-thought approach introduced greater stringency into the scoring process, discrepancies between machine and human grading still existed. The study proposes integrating stringent coefficients into the calculation process to address these remaining inconsistencies. The findings suggest that this integration can further enhance the reliability and consistency of model-generated scores.