Parameter-Efficient Fine-Tuning of Large Language Models Using LoRA and Prompt-Tuning Under Low-Resource Conditions

The process of adapting large pre-trained language models (lms) to tasks with few-labeled data remains a major challenge because of the restrictions on data and computation. This paper mainly discusses two parameter-efficient methods of fine-tuning, Low Rank Adaptation (LoRA) and Prompt Tuning, for binary sentiment classification where the base model is distilbert in low-resource situations. In order to make a fair comparison of the two methods, a controlled experimental setup is used.Experiments have shown that with the size of training data 1,000 LoRA can produce the result 88.7% in terms of accuracy, whereas Prompt Tuning can produce 82.5% but with far fewer trainable parameters (15,360). Even with extremely low-resource conditions (100 samples), LoRA still can have a performance of 70.0%. The results for cross-domain on Yelp review data show that LoRA obtains 86.8% accuracy, whereas Prompt Tuning 84.4%, which is also indicative of a good generalization ability. The result of this work is that LoRA is shown to be a better choice if the performance within the domain is desired, while Prompt Tuning is a better choice in the case of limited parameter budget but will have comparable performance in transfer task. Besides, they shed light on the practical knowledge of choosing fine-tuning methods in resource-constrained environments.