Sentiment Classification in Low-Resource, Code-Mixed Languages: Comparative Study of Llama 3 and Other Large Language Models

Large language models have seen limited evaluation on code-switched languages, despite the increasing use of such linguistic patterns on social media platforms. This study presents a systematic evaluation of Llama 3 for sentiment analysis of "Banglish", the informal blend of Bengali and English widely used online in Bangladesh. We examine five adaptation strategies: zero-shot prompting, few-shot learning, a two-step translation and analysis pipeline, fine-tuning, and model ensembling. Our experiments, conducted on 11,673 posts from the Bengali_Banglish_80K dataset, show that fine-tuning Llama 3 delivers the best performance, achieving an accuracy of 66.87 percent. This result outperforms its own zero-shot (43.80 percent) and few-shot (49.14 percent) baselines, as well as the zero-shot results of GPT-3.5 (55.90 percent), GPT-4 (65.15 percent), and Claude 3.5 (47.68 percent). The dual-phase pipeline achieved 48.07 percent accuracy, and the ensemble method reached 66.78 percent, offering slight improvements but falling short of the fine-tuned model. We release our best-performing model, samiur-r/BanglishSentiment-Llama3-8B, to support further research. These findings highlight the effectiveness of task-specific fine-tuning in low-resource, code-mixed settings and emphasize the need for more comprehensive code-switching datasets and pre-training strategies tailored to linguistically diverse communities.