Knowledge Distillation Framework of Pre-Trained Language Models Combined with Parameter Efficient Fine-Tuning

The development of pre-trained language models(PLMs) has increased the size of models, increasing the demand for computing resources and data when fine-tuning downstream tasks, and challenging deployment in resource-limited environments. In order to alleviate this demand, this paper proposes a knowledge distillation(KD) framework of PLMs combined with parameter efficient fine-tuning(PEFT). By introducing PEFT into both teacher fine-tuning and specific task distillation, the framework reduces computational costs without significantly compromising model performance. The experimental results show that in the GLUE benchmark test, the performance of the student model trained by the distillation framework on multiple NLU tasks is comparable to or even better than that of the teacher, and it shows higher efficiency and robustness in resource utilization.