Comparative Study of LSTM and Transformer Models for Sentiment Analysis on the IMDB 5000 Dataset

Sentiment analysis, the task of determining the sentiment expressed in text, plays a crucial role in various applications such as movie reviews, social media monitoring, and customer feedback analysis. This study presents a comparative analysis of Long Short-Term Memory (LSTM) and Transformer models for sentiment classification using the IMDB 5000 dataset. The dataset consists of 5000 movie reviews, labeled as positive, negative, or neutral, and is widely used for evaluating sentiment analysis models. The objective of this research is to explore the effectiveness of LSTM and Transformer architectures in classifying sentiment, highlighting their strengths and weaknesses in handling text data.The LSTM model, a type of recurrent neural network, is well-suited for processing sequential data and capturing dependencies over time. The Transformer model, on the other hand, leverages self-attention mechanisms to capture relationships across the entire input sequence simultaneously, making it more efficient for handling long-range dependencies. This study evaluates both models on accuracy, F1 score, precision, recall, and AUC, and discusses their performance in terms of computational efficiency, scalability, and robustness against noisy data.Our experiments show that while LSTM excels in tasks with shorter sequences and is more intuitive for sequential data processing, the Transformer model, particularly using pretrained BERT, outperforms LSTM in handling longer sequences and complex sentiment analysis tasks. The Transformer’s ability to capture nuanced semantic information, coupled with its parallel processing capability, allows it to achieve superior results in terms of classification accuracy and generalization.