Leveraging pre-trained self-supervised learning models for low-resource multimodal text and speech sentiment analysis

Multimodal sentiment analysis (MSA) is designed to recognize human sentiment from multiple data modalities, offering richer context and improved accuracy compared to unimodal approaches. The MELD dataset provides text, audio, and visual modalities, which are extracted from episodes of the TV series Friends, with each utterance annotated with both emotion and sentiment labels. This study only utilizes text and speech modalities from the MELD dataset due to the visual features containing a significant amount of background information, which introduced irrelevant noise unrelated to the emotional states of the characters. This approach is particularly well-suited for podcast analysis, as podcast primarily rely on spoken language without consistent visual cues. However, the MELD dataset presents several challenges. These consist of loud audio, feature overlap between sentiment classes, and unbalanced data where neutral sentiments are dominate. To address these issues, a lightweight yet effective model is proposed that integrates BERT-based textual embeddings and Wav2Vec2 for audio features, employing an element-wise fusion strategy followed by a Multi-Head Attention and LSTM (MHA-LSTM) for contextual sequence modeling, which improves representation learning for sequence classification. The MHA-LSTM focuses on relevant input sequences, capturing long-term temporal dependencies. This model achieves competitive performance on MELD, with an overall accuracy of 67.66% and a weighted average F1-score of 0.65, which demonstrate that our approach offers strong accuracy and computational efficiency, making this model suitable for real-world applications where only text and audio modalities are being used.