Beyond looks: the effect of voice personality traits and gestures in virtual agent interactions

Abstract With the expansion of extended reality (XR), virtual agents engage more with humans across fields. While the influence of anthropomorphic traits on user perceptions is studied, the combined effects of voice traits and gestures need further investigation. We explored the impact of a virtual agent's voice personality and gestures on subjective perception and visual behaviors in a hospital guidance setting. Using a 2 × 2 within-subjects design, we assessed the influence of voice traits (calm vs. lively) and gestures (with vs. without) through subjective reports and eye tracking. Results show that the impact of gestures on content comprehension ease varied depending on the voice personality traits. Specifically, a lively voice with gestures significantly increased comprehension while a calm voice with gestures did not. Additionally, a calm voice led to longer average fixation durations on body-related areas of interest (AOIs), shorter time to first fixation, and fewer fixation counts on dialog content-related AOIs. Gestures decreased fixation counts on face-related AOIs. For some eye metrics, the impact of gestures varied depending on voice personality traits, whereas for others, the influence of voice traits depended on the presence of gestures. Thus, voice traits and gestures influence user perceptions and visual attention differently.