Research on the Lightweight of Natural Language Understanding Model in Electric Power Field

To address the application requirements of indicator question-and-answer (Q&A) in the electricity industry, this paper researches a natural language understanding model for balancing performance, computing resource consumption, and inference time, and proposes a method to incorporate an early stopping mechanism in the lightweight process of knowledge distillation. When training the student model with the knowledge of the teacher model, patient early stopping mechanism is added to each layer of encoding, allowing samples of different complexity to be encoded with different layers, reducing overfitting problems that are prone to occur in the student model, and controlling the inference time of the model. On the dataset constructed by the real data in the power industry, experiments show that when the parameters of the student model are reduced to 60% of the teacher model, and the inference time is reduced by nearly 50% , while the accuracy of the model is only reduced by about 2% , maintaining high availability. Compared with the benchmark models, it achieves a better balance in performance and inference time.