I Talk - You Write. Exploring Asymmetric Text- and Voice-Based Communication Between Humans and Virtual Agents

The increasing relevance of artificial intelligence (AI) agents for people’s everyday professional and private lives leads to a great need for identifying efficient and user-friendly modalities for human-agent interaction. Such interaction usually takes place via text or voice. While chatbots and voice assistants each rely on only one modality, the focus of this paper is on asymmetric modality combinations. A user study was conducted in the context of information retrieval, and different input and output modalities were evaluated in a 2 × 2 within-subject design. It was found that efficient modality combinations are not always preferred and that usability factors, such as user control, have a greater influence on preference. For example, the combination of voice input and text output can prove to be very efficient, but at the same time be less preferred due to weaknesses in the user experience. Pure text-based interaction was found to be the preferred combination as it ensures a high degree of control and freedom for the user. Dialog-based applications for information retrieval should therefore use the text modality as a basis and supplement this with voice-based input and output options.