Multimodal Emotion Recognition in Empathetic Dialogue: An Integrated Approach Using Speech and Text

With the advancement of computer performance and the progress in machine learning and natural language processing technologies, interactive AI systems capable of dialog with humans have emerged. However, in achieving truly human-like interaction, the ability to understand human emotions and exhibit appropriate empathetic responses, rather than merely providing logical replies, is still room for improvement. In this study, we have constructed models using Japanese empathetic spoken dialog corpus that includes empathetic conversations. We have developed models that utilize speech features, text features, and a multimodal model that integrates both. Comparing the accuracy of each model, the speech model achieved 0.752, the text model achieved 0.704, and the multimodal model achieved 0.813, with the multimodal model demonstrating the highest accuracy. By comparing the accuracy of each model by emotion, we have identified which model is more effective in recognizing specific emotions and demonstrated that the multimodal model effectively captures diverse data features, contributing to improved accuracy.