Seeing Meaning: How Congruent Robot Speech and Gestures Impact Human Intuitive Understanding of Robot Intentions

Abstract Social communication between humans and robots has become critical as a result of the integration of robots into our daily lives as assistants. There is a need to explore how users intuitively understand the behavior of a robot and the impact of social context on that understanding. This study measures mental effort (as indexed by pupil response) and processing time, measured as the time taken to provide the correct answer, to investigate participants’ intuitive understanding of the robot’s gestures. Thirty-two participants participated in a charades game with a TIAGo robot, during which their eyes were tracked. Our findings show a relationship between mental effort and processing time, and indicate that robot gestures, congruence of speech and behavior, and the correctness of interpreting robot behavior influence intuitive understanding. Furthermore, we found that people focused on the robot’s limb movement. Using these findings, we can highlight what features contribute to the intuitive interaction with a robot, thus improving its efficiency.