EAC-Agent: A deep learning framework for multimodal emotion-aware conversational agent with contextual response generation

Conversational agents, commonly referred to as chatbots, have become an integral part of various important applications, such as customer support and virtual assistants in enterprise-level solutions. Despite their widespread use, current solutions predominantly rely on a single modality input, primarily text, which limits their ability to fully understand the correct emotions of users and respond accordingly. The increasing demand for multimodal inputs in conversation such as text, audio, and videos, highlights the need for a more comprehensive approach. Existing conversational agents face significant challenges in generating emotionally aware responses, as they lack the ability to effectively handle emotion embeddings, leading to limitations in emotional accuracy and contextual appropriateness. Focusing on these challenges, this research work leads to a novel multimodal approach which incorporates features from text, audio, and visual. The EAC-Agent proposed a sequence-to-sequence model with transformer, along with pre-trained embeddings such as GloVe. The self and cross-modal attention on text, audio and visual is used to generate a more emotionally intelligent response. EAC-Agent is validated through comparison with existing techniques on two benchmark datasets. The obtained results demonstrate superior performance in emotion classification and response generation. The proposed model achieves an accuracy of 76.27% on IEMOCAP and 67.57% on MELD for emotion recognition from multimodal user inputs. In addition, the emotion-aware response generation module shows clear improvements, with perplexity values of 39.01 and 42.30, BLEU scores of 0.31 and 0.30, and ROUGE-L scores of 0.45 and 0.44 on IEMOCAP and MELD, respectively. EAC-Agent demonstrates clear superiority over existing models and holds great promise for applications in customer service, healthcare, and other areas requiring empathetic and contextually appropriate interactions.