Multilevel Attention-based Multimodal Emotion Recognition from Text and Audio

Automatic emotion recognition has been a longstanding challenge in the field of affective computing. Within this domain, Speech Emotion Recognition (SER) has gained increasing attention as a promising application of machine learning. The primary objective of SER is to enhance emotion detection by leveraging specific types of features (such as MFCC and spectral features) or by combining multiple feature types. However, current approaches that rely solely on individual acoustic features face difficulties in fully exploiting the rich information embedded in speech signals. To overcome this limitation, we introduce a novel emotion recognition system based on a multi-level attention mechanism, which integrates both high-level textual features and multi-layered audio features for more effective emotion classification. By incorporating information from diverse modalities, this approach aims to improve recognition accuracy, addressing the shortcomings of single-modality systems. Experimental evaluations on the IEMOCAP and CASIA datasets confirm that our method successfully captures the intricate relationships between multi-level features, enabling refined information processing and significantly boosting model performance, thus providing a strong competitive edge.