EmoMATE: A Multimodal Emotion-Aware AI Assistant with Trust-Adaptive Interaction Modeling

In recent years, the integration of multimodal interaction and affective computing has driven a paradigm shift in human-AI collaboration. This paper proposes EmoMATE, a novel affect-aware AI assistant framework that dynamically fuses visual, auditory, and linguistic modalities to perceive and respond to user emotions in real-time. Unlike traditional assistant systems that rely on static rule-based logic, EmoMATE leverages a hybrid transformer architecture with cross-modal attention alignment and a continual trust calibration module to co-model affective cues and trust dynamics. We introduce a trust-aware reinforcement learning strategy that adaptively adjusts the assistant's communicative behavior based on inferred user trust states. Through a longitudinal user study involving 96 participants across diverse contexts (productivity, emotional support, decision-making), we demonstrate that affect-driven interaction significantly improves perceived competence, empathy, and trust retention. Our findings reveal that multimodal emotional responsiveness is not only critical for user engagement but also acts as a core driver for long-term trust formation in AI systems. This work bridges affective computing and trust modeling, providing new insights into the design of socially intelligent, emotionally adaptive AI assistants.