Power engineering domain pretrained language model with natural language processing benchmarks

Interest in applying artificial intelligence to domain-specific problems has been steadily increasing. Power engineering, a critical subdiscipline of electrical engineering, has so far seen relatively limited integration of advanced natural language processing (NLP) methods. To address this gap, we introduce Power Engineering BERT (PoE BERT), a domain-specific model based on Bidirectional Encoder Representations from Transformers (BERT), pretrained on a specialized corpus of one million open-access power engineering paper abstracts. In addition, we present the first set of NLP benchmarks tailored to power engineering, covering tasks such as document classification, question answering, and named entity recognition. Experimental results demonstrate that PoE BERT achieves an average performance improvement of approximately 13% compared with BERT-small while using a similar number of parameters. These findings underscore the importance of domain-specific pretraining for enhancing NLP performance in power engineering and highlight the contributions of this study in providing both a novel model and benchmark resources for future research in this domain.