Advancing Sentiment Analysis with LLM: Comparative Study with Traditional ML on Social Media Reviews

This research develops a sentiment classification model using machine learning to analyze textual data. It follows three stages: data collection, preprocessing, and model development. Using the NLTK dataset, text mining techniques extract and refine features. Supervised machine learning classifies tweets as positive or negative, with testing across two frameworks yielding higher accuracy than prior studies. Given Twitter’s prominence, this study examines whether sentiment analysis models can match human evaluators in educational research. Sentiment ratings from 333 students on their learning experiences were analyzed. Nine machine learning models were tested under five experimental setups, alongside two non-ML methods. Comparisons with human raters’ sentiment ratings revealed that the Naïve Bayes model achieved a 98% accuracy rate after excluding neutral sentiments. Challenges in identifying neutral sentiments were noted, with a word-sentiment association method attaining 87% accuracy without model training, enhancing generalization and adaptability. Expanding educational datasets is crucial for improving sentiment analysis models, as AI and machine learning facilitate automated classification. Businesses leverage this technology to analyze consumer emotions, monitor social media discussions, evaluate reviews, assess employee sentiment, and detect harmful content. Traditional models like SVM excel in processing short text, while GPT-4 outperforms them in analyzing context-rich texts, achieving superior precision, recall, and F1 scores. Additionally, GPT-4 is more effective in detecting mixed sentiments. Findings suggest that large language models enhance sentiment classification accuracy, providing valuable insights for businesses extracting meaningful information from textual data.