HGLER: A hierarchical heterogeneous graph networks for enhanced multimodal emotion recognition in conversations

This research has proposed a new Emotion Recognition in Conversation (ERC) model known as Hierarchical Graph Learning for Emotion Recognition (HGLER), built to go beyond the existing approaches that find it difficult to request long-distance context and interaction across different data types. Rather than simply mixing different kinds of information, as is the case with traditional methods, HGLER uses a 2-part graph technique whereby conversations are represented in a 2-fold manner: one aimed at illustrating how various parts of the conversation relate and another for enhancing learning from various types of data. This dual-graph system can represent multimodal data value for value by exploiting the benefits of each type of data yet tracking their interactions. The HGLER model was applied to two widely used datasets, IEMOCAP and MELD, with many varieties of information, texts, pictures, or sounds, hence, to see to what extent the model can understand emotions in conversations. Preprocessing methods common in practice were done to make things consistent, and the datasets were set aside for training, validation, and testing informed by previous works. The model was tested using two standard datasets, including IEMOCAP and MELD. On IEMOCAP, HGLER posted an F1-score of 96.36% and accuracy of 96.28%; on MELD, it posted an F1-score of 96.82% and accuracy of 93.68%, surpassing some state-of-the-art methods. The model also showed some superb performance in terms of its convergence, generalization, and convergence stability during training. These findings demonstrate that hierarchical graph-based learning can be applied in enhancing emotional comprehension in systems dealing with several forms of information in handling conversations. However, slight changes in validation loss observed suggest there are areas of model stability and generalization to be improved on. These results validate that using hierarchical graph-based learning in multimodal ERC does well and promises to enhance emotional understanding in conversational AI systems.