A review of multimodal sentiment analysis: Taxonomy, issues, challenges, and future perspectives

Recent developments in computational intelligence have produced a huge volume of multimodal data across different digital platforms. This data is a great source of contextual, sentimental, and emotional information. Multimodal sentiment analysis (MMSA) is the process of inferring sentiments from multimodal data. MMSA has improved the effectiveness and accuracy of sentiment analysis by integrating heterogeneous modalities. However, there are several issues and challenges in combining multiple modalities, like high complexity, modality fusion, lack of explainability, and temporal synchronization. This paper presents a review of MMSA, discussing data modalities, fusion approaches, issues and challenges. It also presents the statistical analysis and overview of datasets and evaluation metrics used in the reviewed papers. Moreover, it identifies several future research opportunities for the research advancements in MMSA. It is believed that the article will be beneficial for the researchers working in the relevant field. • Comprehensive review of multimodal sentiment analysis with key issues, challenges, and solutions. • Analyse the datasets and evaluation metrics in multimodal sentiment analysis. • Presents a compact framework leveraging early fusion for multimodal sentiment analysis. • Future research directions offering insights for multimodal sentiment analysis researchers.