A Study on the Application of an Improved Fine-tuning Method based on LoRA with Transition Matrix for Efficient Parameter Fine-tuning of Large Models

With the rapid development of large language models (LLMs), efficient parameter fine-tuning has become increasingly important, enabling ordinary researchers to utilize the functions of LLMs in personalized and specialized tasks. The latest progress has proven that various efficient parameter tuning methods can achieve the effect of full parameter training. These methods have significantly reduced the resource consumption of model training. However, how to maintain the generalization ability of large models while improving the fine-tuning effect still remains a challenge. Addressing this issue, this paper thoroughly investigates various mainstream fine-tuning methods, such as LoRA and Ptuning v2, and analyzes their shortcomings from a model mechanism perspective. Then, an improved fine-tuning method based on LoRA is proposed. Our method ensures that the input feature dimension is gradually reduced by adding an intermediate matrix, so that the model can capture key information effectively. Comparative analysis on the public dataset ceval shows that the effectiveness of the proposed method is confirmed.