Evaluating parameter-efficient fine-tuning of large language models for sentiment analysis with bayesian optimization

The increasing volume of user-generated reviews and feedback available in digital format provides valuable insights across various domains. Sentiment analysis plays a crucial role in extracting opinions from textual data. Large Language Models (LLMs) have demonstrated exceptional performance in numerous Natural Language Processing tasks; however, fine-tuning these models remains resource-intensive and time-consuming. Parameter-Efficient Fine-Tuning (PEFT) techniques offer a more computationally efficient alternative. This study investigates the effectiveness of Low-Rank Adaptation (LoRA) for fine-tuning LLMs in Arabic sentiment analysis by evaluating both sequence classification and instruction-tuning paradigms across the ASTD, LABR, and HARD datasets. The results show that LLaMA-3-8B-Instruct achieves competitive performance compared to LLaMA-3-8B with a classification head. Moreover, SILMA-9B-Instruct shows the highest overall performance, achieving F1-scores of 88.95%, 89.91%, and 96.55% on ASTD, LABR, and HARD respectively. In addition, Bayesian hyperparameter optimization is employed to find optimal LoRA configurations by exploring rank, dropout, and learning rate so that maximum validation scores can be achieved with respect to the F1-score for the entire model.