A Hierarchical Attention-Based Fusion Model for Multimodal Sentiment Analysis of Customer-Generated Review Videos

Videos have become a common medium for customers to share their feedback, driven by the widespread use of the internet and the resulting proliferation of diverse content across social media platforms. These videos often contain multimodal forms of data, such as text, audio, and visual components, making them rich sources for sentiment analysis (SA). Multimodal sentiment analysis (MSA) combines data from all these modalities to improve understanding of sentiments and achieve an accurate prediction. However, existing multimodal fusion methods do not perform satisfactorily well, as they either concatenate raw features or often overlook intricate interdependencies between modalities, failing to resolve conflicts between them. This work attempts to implement a novel fusion method, Hierarchical Attention-Based Fusion (HABF), which uses self-attention and cross-attention mechanisms in a hierarchical manner that prioritizes and integrates efficient features coming from all modalities. HABF combines unimodal features by assigning contextual weights to each modality, ensuring the most accurate representation of sentiments in videos. The support vector machine then classifies the fused representation into positive, neutral, or negative sentiments. The model is tested on two datasets: (1) the Customer-Generated Sentiment Videos (CGSV) dataset, which is restaurant review-based, and (2) the standard dataset, CMU-MOSI. The proposed model uses Bidirectional Long Short-Term Memory (BiLSTM) for text, Librosa for audio, and a Convolutional Neural Network (CNN) for visual data for feature extraction. The evaluated results are compared with existing systems using CMU-MOSI, achieving better accuracy with 82.43%. The proposed model shows an accuracy of 78.10% using the CGSV dataset. This study focuses mainly on improving MSA for customer satisfaction, dining experience, and service quality, and subsequently increasing better business strategies for restaurants.