Attention-guided Automatic Scoring with Large Language Models

Traditional models often fail to provide an efficient solution to the complex scoring of subjective questions. In the present work, an attention-guided automatic scoring (AGAS) method with large language models (LLMs) was proposed to address this problem. Based on the outstanding performance of LLMs in language understanding and reasoning and the scoring procedures for subjective questions, a set of template prompts were designed to “guide the attention” of the LLM to fulfill the scoring task. Meanwhile, inspired by the notions of chain-of-thought and the few-shot prompting technique, we performed comparative experiments on 19 online and offline LLMs. It was found that models with AGAS achieved better scoring quality and accuracy, with a 40.5% improvement on offline models, which verifies the effectiveness of the AGAS method and its better performance in efficiency and accuracy than traditional methods. The proposed method has been successfully applied to the examination & training system in a renewable energy company, where it has achieved good performance.