Knowledge Distillation and Parameter Simplification for Power Operation and Maintenance-based LLMs

The large language models (LLMs) have great potential for application in the field of power operation and maintenance. Unlike previous single visual models, the visual big model has shown great advantages in image understanding with its massive multi-modal information and huge network parameters. However, due to the huge number of parameters in the current general visual big models, it is difficult to deploy them on site and there are few practical applications. Therefore, this article proposes knowledge distillation and model pruning for power-specific models. After general visual big models and data processing, high-quality pseudo-label data is obtained, and knowledge distillation is used to obtain a student model with fewer parameters. Then, the student model is sparsely trained and pruned, and fine-tuned on the power data set to obtain an easy-to-deploy lightweight power-specific model. Finally, a student model with a parameter volume of 100 million to 1 billion is obtained through knowledge distillation. Through model pruning, the model is compressed to more than 50% of the volume before pruning, and the model detection rate drops by no more than 1%, effectively improving the efficiency of the model.