ElecBench: A large language model benchmark in electric power domain

Large language models (LLMs) have made substantial advancements in the field of natural language processing, necessitating the development of new benchmarks to accurately track their progress. In this paper, we introduce ElecBench, the first benchmark specifically designed for the electric power domain. ElecBench comprises 24 datasets spanning different scenarios, covering general electric power knowledge and four specific business applications, with a total of 34,030 data entries. Furthermore, we evaluate the performance of a series of open-source Chinese LLMs on ElecBench. Our experiments demonstrate that ElecBench serves as an effective benchmark for electric power scenarios and highlight that existing LLMs require further optimization to gain domain-specific knowledge and achieve better performance. • ElecBench: the first large language model benchmark for electric power domain. • Covers 24 datasets across general knowledge and four business applications. • Evaluates multiple open-source Chinese large language models on the benchmark.