Comparative Study of Llama and Bert for Sentiment Analysis on Stock-Related Social Media and Its Impact on Price Prediction

With the growing impact of retail investors and online communities on financial markets, sentiment analysis of social media has become an increasingly valuable tool for stock price forecasting. This study investigates the effectiveness of two state-of-the-art transformer-based language models-BERT and LLaMA-for sentiment analysis on Stocktwits messages related to three U.S. equities: AAPL, AMD, and TSLA. We collected Stocktwits messages from July 1, 2010, to December 31, 2024, and classified each message as positive or negative using both models. These sentiment labels were combined with historical stock price features-Open, High, Low, and Close (OHLC)-to create multimodal input vectors. A Long ShortTerm Memory (LSTM) neural network was then used to predict the next-day closing prices based on a 7 -day sliding window of both sentiment and price data. Model performance was evaluated using Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{R}^{\mathbf{2}}$</tex>. Results show that incorporating sentiment significantly improves predictive accuracy over price-only models. Additionally, LLaMA outperformed BERT in sentiment classification, leading to better forecasting performance for several of the target stocks. These findings highlight the promise of integrating transformerbased sentiment analysis with deep learning models for enhanced financial market prediction.