Evaluation and Analysis of Large Language Models Performance in English Exam

This study aims to evaluate the capabilities and risks of Large Language Models (LLMs) in English exam. The paper collected a dataset comprising 850 real exam questions to assess the answer performance of LLMs represented by GPT-4, SparkDesk v3.5, ERNIE 4.0, Claude-3, and GLM-4. To achieve this, the study conducted two rounds of comparative experiments. Firstly, this study assessed the ability of LLMs to generate accurate answers under 0-shot using precision metric and BLEU score. Subsequently, the study evaluated the quality of explanations provided by LLMs, using questionnaires, in comparison to those provided by human experts. In the second round of experiments, the paper explored the impact of various prompts on models' performance, and measured the stability of their outputs using confidence metric. The results indicates that: (1) Most LLMs achieve high accuracy in answer generation, particularly GPT-4, excelling in objective multiple-choice questions with an accuracy exceeding 90%. (2) Regarding explanation quality, a gap exists between LLMs and human experts, notably in areas of knowledge storage and logical reasoning. (3) In terms of the assistance of prompts, employing specific prompting techniques can affect the performance of LLMs, with "3-shot-Chain-of-Thought" and "Self-Consistency" showing the most significant improvement. (4) Regarding output stability, LLMs generally perform well, especially Claude-3, which achieves nearly perfect consistency. Based on these findings, the paper proposes recommendations for the application of LLMs in the field of English education, aiming to promote LLMs to serve educational practice more effectively.