Multimodal Sentiment Analysis Approach Combining Transfer Learning and Generative AI of E-Commerce Product Reviews

The rapid growth of e-commerce in Indonesia, with Tokopedia as a leading platform, generated an abundance of online reviews containing both text and images, which offered valuable insights into customer experiences. These reviews provided essential feedback that helped businesses understand consumer sentiment, preferences, and expectations. Traditional sentiment analysis models like BERT, RoBERTa, and FNet, performed well with text data but struggled with visual information. This research introduced a hybrid approach that combined Transfer Learning for text analysis with a Convolutional Neural Network (CNN) for image analysis to enhance overall multimodal sentiment analysis accuracy. When visual content was unavailable, Stable Diffusion v2-base can generated synthetic images from text, which significantly improved the availability of visual data. Following the Knowledge Discovery in Databases (KDD) framework, which included selection, preprocessing, transformation, data mining, and evaluation, this study found that BERT achieved the highest text-based accuracy at 90.97 % and RoBERTa at 90.33 %. In the image-based approach, CNN reached an accuracy of 54.25 %, while the hybrid CNN-BERT model led with <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{9 0. 6 0 \%}$</tex>, followed by CNN-RoBERTa at <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{8 9. 4 0 \%}$</tex>. These findings highlighted the potential of multimodal sentiment analysis in providing effective marketing strategies, informing product development, and enhanced customer satisfaction, ultimately offering a significant competitive advantage in the e-commerce sector.