Multimodal Emotion Fusion Mechanism and Empathetic Responses in Companion Robots

The ability of humanoid robots to exhibit empathetic facial expressions and provide corresponding responses is essential for natural human–robot interaction. To enhance this, we integrate the GPT3.5 model with a facial expression recognition model, creating a multimodal emotion recognition system. Additionally, we address the challenge of realistically mimicking human facial expressions by designing the physical structure of a humanoid robot. Initially, we develop a humanoid robot capable of adjusting the positions of its facial organs and neck through servo displacement to achieve more natural facial expressions. Subsequently, to overcome the current limitation where emotional interaction robots struggle to accurately recognize user emotions, we introduce a coupled generative pretrained transformer (GPT)-based multimodal emotion recognition method that utilizes both text and images, thereby enhancing the robot's emotion recognition accuracy. Finally, we integrate the GPT-3.5 model to generate empathetic responses based on recognized user emotional states and language text, which are then mapped onto the robot to enable empathetic expressions that can achieve a more comfortable human–machine interaction experience. Experimental results on benchmark databases demonstrate that the performance of the coupled GPT-based multimodal emotion recognition method using text and images outperforms other approaches, and it possesses unique empathetic response capabilities relative to alternative methods.