VR & in-lab GESs: an analysis of empirical research designs for gesture elicitation in in-lab and virtual reality settings

Abstract Empirical study designs in HCI evolve in response to temporal realities and technological advancements. In this context, virtual reality (VR) shows potential for new empirical research designs, going beyond the (still) quite dominantly lab-based research roots in HCI. Previous work has been conducted to identify the use of VR for gathering non-homogeneous and representative sample populations and for conducting empirical studies in resource-constrained environments. Yet, it is unclear how VR empirical user study designs affect the participants’ behavior and experience, potentially influencing the study results compared to in-situ/in-lab studies. In this paper, we conducted a gesture elicitation study (GES) in a realistic physical smart room and its digital duplicate in VR. Sixty-six participants’ responses were collected using standardized questionnaires along with between-group gesture agreement analysis. Our comparison shows that the VR study produces a higher number of unique gesture proposals and similar best gestures to the in-person study for 95.4% of the referents, with minimum influence on the gesture proposals. We further discuss the usability, pragmatic and hedonic qualities, presence, task load, and implications of using VR for GESs, and highlight future directions for using VR-based empirical study designs. We found that VR can produce reliable data and improve participant experience with the same task load, making it viable to conduct remote GES and a substitute for conventional lab-based experiments.