PHILIA: Emotionally Aware Multimodal Conversational Agent

Human communication is naturally multimodal and involves speech, facial expressions, and language context to express emotions effectively. Unfortunately, most current conversational AI systems are emotion-insensitive or unimodal, resulting in emotionally inappropriate or insensitive responses. This paper describes the design and development of a real-time Emotion-Aware Multimodal Conversational Agent that combines speech, facial, and text modalities to infer the user’s emotional state and respond with emotionally appropriate messages through text-to-speech. Our proposed system uses deep learning-based speech emotion recognition, Vision Transformer for facial emotion recognition, and transformer models for text emotion analysis. A confidence-weighted multimodal fusion approach is adopted to effectively combine emotional information from multiple modalities, even with noisy or missing data. The inferred emotional state is then used to condition a large language model for adaptive response generation, followed by emotion-aware speech synthesis for expressive interaction. The modular design is scalable, real-time, and easy to deploy. Experimental results with standard datasets and real-time demonstrations demonstrate the efficacy of the proposed approach in improving empathetic and humancentric conversational AI systems.