Improving Perceived Emotional Intelligence of Embodied Chatbot Haru via Multi-Modal Interaction

To realize natural and friendly interaction, this paper proposed an emotion-aware framework to enable embodied chatbot Haru to mimic human users' emotions via multi-modal interaction during conversation. The proposed framework consists of ASR for speech recognition, ChatGPT for dialogue generation and Text-to-Speech for text to speech conversion. In addition, the pitch, rate and emotions of human speech detected with deep learning models are used to adjust the pitch and rate of Haru's speech. In addition, Haru can mimic the emotional state of human users by expressing emotive routine behaviors consisting of base rotation, eye animation, mouth movement etc., based on the detected emotion of human speech. Results of a user study with 20 subjects show that our embodied chatbot Haru can successfully mimic the intonation and emotion of human speech via vocal and visual interactions. Moreover, participants reported to have a better conversing experience with our embodied chatbot Haru compared to a neutral one with only lip synchronization.