Talking surveys: How photorealistic embodied conversational agents shape response quality, engagement, and satisfaction

Embodied conversational agents (ECAs) are increasingly more realistic and capable of dynamic conversations. In online surveys, anthropomorphic agents could help address issues like careless responding and satisficing, which originate from the lack of personal engagement and perceived accountability. However, there is a lack of understanding of how ECAs in user experience research may affect participant engagement, satisfaction, and the quality of responses. We introduce a method, Virtual Agent Interviewer, and validate it in a randomized study. Our proof-of-concept method enables the incorporation of conversations with a virtual avatar into surveys using AI-driven video generation, speech recognition, and Large Language Models. In our between-subjects study, 80 participants (UK, stratified random sample of the general population) either talked to a voice-based agent with an animated video avatar, or interacted with a chatbot. Our evaluation entails 2265 conversation responses obtained across surveys based on two self-reported psychometric tests. Statistical comparison of the results indicates that embodied agents can contribute significantly to more informative, detailed responses, as well as higher yet more time-efficient engagement. Furthermore, qualitative analysis provides valuable insights about the causes of no significant change to satisfaction, linked to personal preferences, turn-taking delays, and Uncanny Valley reactions. These findings support and inform the development of new AI-driven embodiment-based methods for the transformation of online surveys into more natural interactions resembling in-person interviews.