From Lexicons to Transformers: A Comprehensive Analysis of IMDB Sentiments

Sentiment analysis is a critical area in natural language processing (NLP) for determining and extracting subjective textual information. Sentiment analysis aims to determine whether a text states a positive, negative, or neutral view. In the modern digital era, an increasingly large volume of user-generated content, including movie reviews, tweets, and product comments, offers a window into the public perception. Sentiment analysis enables organizations to make data-driven decisions, monitor brand sentiment, and enhance user experience. This study investigates sentiment analysis on the IMDB movie reviews dataset using three primary approaches: lexicon-based methods, traditional machine learning (ML) models, and transformer-based deep learning (DL) models. We conduct a comprehensive comparison to evaluate each method’s effectiveness in capturing sentiment. Lexicon-based techniques like VADER and TextBlob offer interpretable, rule-based insights, while ML models such as logistic regression and support vector machines leverage engineered features for sentiment prediction. Transformer models, particularly BERT, demonstrate superior performance by capturing deep contextual information, achieving an accuracy of 93%. Additionally, we propose a hybrid model that integrates RoBERTa embeddings with lexicon-based features and a custom neural network, attaining a competitive accuracy of 91%. Our findings highlight the advantages and trade-offs of each approach, underscoring the potential of hybrid models in enhancing sentiment classification performance. This work contributes to advancing sentiment analysis methodologies for real-world applications in entertainment and beyond.