Multimodal Sentiment Analysis for the Malay Language: New Corpus using CNN-based Framework

Sentiment analysis in the Malay language has traditionally focused on text-based data. Malay is the native language of Malaysia and other surrounding countries. While text-based sentiment analysis has shown good performance, it often lacks accuracy due to the absence of the speaker's affective state and intentions, which can lead to misinterpretations. Multimodal sentiment analysis addresses these shortcomings and has demonstrated improved performance in various prediction tasks. Unfortunately, there has been little research in this area for the Malay language, due to a lack of corpus and baseline studies. This paper introduces a new Malay Multimodal Sentiment Corpus, ‘MyMSC’, with annotations at both the multimodal and unimodal text levels. It contains 1,208 segments covering political and social topics. The corpus development processes are described in detail, along with the necessary guidelines and considerations. This paper proposes a CNN-based framework with a late fusion method as the baseline model. Experiments with the proposed model demonstrate that the multimodal approach (F1 score = 0.77) outperforms the unimodal approach (F1 score = 0.68), validating the contribution of multimodality to classification performance. The differences between the two types of annotations and their impact are further elaborated. The full corpus is available at https://github.com/serenaroseaini/MyMSC .