Automated Annotation of Academic Emotion Intensity in Online Learning Comment Texts: A BWS Method Based on LLMs

Academic emotions significantly impact learning processes and student performance, with a recent trend towards automated measurement for their types and intensity. However, manual annotation methods for large-scale training data required by modeling face issues of time consumption and high cost. The Best Worst Scaling (BWS) methodology enhances the reliability of intensity annotation, while Large Language Models (LLMs) offer advantages in understanding academic emotions across diverse contexts. Combining the BWS and LLMs in academic emotion intensity annotation, this study aims to address the challenge of data annotation in measuring academic emotion intensity in online learning. We choose three widely recognized LLMs to complete the BWS annotation tasks separately, then calculate the consistency and conduct statistical analysis. Results indicate that the consistency of the three LLMS in identifying emotion intensity in nine academic emotions was above 0.750, with a total of 0.865 in 4569 comment texts. The perception of emotion intensity by the LLMs closely resembles that of human cognition and responds to the context of online learning, enabling them to effectively substitute for humans in performing large-scale annotation tasks.