Deep cross-modal affective memory networks with adaptive multi-source heterogeneous transfer learning in speech emotion recognition

An innovative Deep Cross-Modal Emotional Memory Network (DCM-EMNet) and an Adaptive Multi-source Heterogeneous Transfer Learning Framework (AMS-HTLF) are proposed in this paper. The multimodal data fusion and multi-source heterogeneous data migration problems in speech emotion recognition are effectively solved by this method.In DCM-EMNet, multi-level feature fusion, dynamic affective memory mechanism and cross-modal consistency constraints are adopted to make full use of the bimodal information of speech and text descriptions, and the accuracy of emotion recognition is thereby significantly improved. Meanwhile, through the AMS-HTLF framework, adaptive feature alignment and heterogeneous label mapping are implemented, heterogeneous data from multiple different source domains are effectively migrated, and the generalization ability of the model on the target domain is significantly enhanced. Experimental results show that significant performance improvement is achieved by the proposed method on multiple speech emotion recognition datasets, and its effectiveness and practicality are fully verified. This study not only provides new research perspectives and methods in the field of speech emotion recognition, but also expands new ideas in the field of multimodal learning and transfer learning.