Fine-Tuning Large Language Models for Sentiment Classification of AI-Related Tweets

Fine-tuning large language models (LLMs) for sentiment analysis presents a promising avenue for enhancing the understanding of public sentiment surrounding artificial intelligence (AI) topics. This study explores the fine-tuning of Large Language Models (LLMs) for sentiment analysis on AI-related topics using social media data. We collected approximately 20,000 tweets containing hashtags and comments related to AI concerns, including ‘AI Danger,’ ‘AI Ethics,’ ‘AI Problems,’ ‘AI Creating Problem on Jobs,’ and ‘AI Threat.’ The dataset captures public sentiment surrounding potential negative impacts of artificial intelligence. Three cutting-edge LLMs, Llama 2, Gemma, Mistral, and Phi 3, served as our foundational models for fine-tuning. To enhance model performance, we implemented advanced prompt engineering techniques, including Chain-of-Thought (COT), Algorithm-of-Thoughts (AOT), Fewshot and EmotionPrompt. Our results demonstrate that Llama 2 achieves consistent performance across all techniques with accuracies ranging from 82.3% to 83.9%. Phi-3 showed the best performance with EmotionPrompt and Fewshot techniques, achieving accuracies of 87.8% and 87.4% respectively. Gemma 7B performed optimally with Fewshot at 84.0% accuracy, while Mistral achieved 85.0% accuracy using the same technique. The research reveals that Fewshot and AOT consistently produce reliable results across models, with Emotion-Prompt showing model-specific effectiveness. Our findings have applications for enhancing the precision and dependability of sentiment analysis instruments that track public opinion and AI-related speech.