Unboxing Manipulation Checks for Voice UX

Abstract Voice-based interaction is experiencing a second wind through the advent of machine learning (ML) techniques, affordable consumer products and renewed work on natural language processing (NLP) and large language models (LLMs). A growing body of work is exploring how users perceive new forms of computer-generated voices from qualitative and quantitative angles. However, critical voices have called for greater rigour, especially in confirming the voice as a manipulated variable, i.e. manipulation checks. We present three case studies that highlight the value of investing in rigorous manipulation checks for HCI researchers and designers. We demonstrate the importance of testing assumptions, the need for care and reflection in the design of response options and measurement and the advantages of more exploratory approaches to understanding user perceptions of and user experiences (UX) with voice phenomena. Through these case studies, we raise awareness, empirically justify and critically assess the value of manipulation checks for voice UX research and beyond.