BERT-DeiT Model: Multimodal Sentiment Analysis for E-Commerce Product Review

The rapid development of e-commerce had driven an increase in the number of customer reviews in the form of text and images, but many reviews were incomplete, consisting of only one modality, which made it difficult for companies to understand customer opinions comprehensively. The conventional sentiment analysis approach that relied solely on text or images separately was less capable of capturing the complex multimodal relationships in customer reviews. Therefore, this research developed a multimodal sentiment analysis method by integrating text and image features to enhance the understanding of customer preferences. Transfer learning-based transformer models, namely Bidirectional Encoder Representations from Transformers (BERT), A Lite Bidirectional Encoder Representations from Transformers (ALBERT), and eXtra Long Neural Network (XLNet), were used for text analysis, while Vision Transformer (ViT) and Data-efficient Image Transformer (DeiT) were applied for image analysis. To address the limitation of the dataset that lacked images, the Stable Diffusion v2-base model was used to generate image representations based on review texts. Feature integration from the best models was performed using the concatenation technique, and the entire experiment followed the Knowledge Discovery in Databases (KDD) framework, which included the stages of selection, preprocessing, transformation, data mining, and evaluation. The experimental results showed that the multimodal approach with the combination of the best models, namely BERT for text and DeiT for images, achieved the highest accuracy of 93.49% compared to single models. These findings proved that the integration of text and image modalities could improve classification accuracy compared to unimodal models, thereby providing more comprehensive insights for e-commerce companies in understanding customer sentiment and enhancing the quality of their products and services.