Exploring user feedback in VR: the added value of qualitative evaluation methods

Evaluating immersive experiences, such as Virtual Reality (VR), is often done through User Experience (UX) and Quality of Experience (QoE) evaluations. Here, methodologies such as conducting experiments, the use of questionnaires or data logging can provide valuable insights into implicit user behaviour in their interactions with interactive media experiences (IMX). These quantitative methods can be combined with qualitative methods for user evaluation purposes, making it a mixed methods approach. Consulting literature, extensive research has been done to discover the various benefits and pitfalls of incorporating a mixed methods approach in evaluation studies. However, there is limited research on the added value of a qualitative research methodology in VR user evaluation studies and how these qualitative findings can provide novel insights. This paper presents a mixed-method user evaluation study of VR Planica, a novel VR experience developed by HSLU together with AFP, RTVSLO, Sparknews and VUB as part of the EU project TRANSMIXR. We begin by outlining the conceptual differences between QoE and UX, as well as the conceptual outline of mixed method approaches for user evaluation. Next, we describe our evaluation set-up, and our mixed methods approach applied in the VR Planica user evaluation study involving 33 participants. Drawing on the qualitative data, we reveal insights into how participants experienced and interpreted the experience, which helped deepen the interpretation of the quantitative results. We conclude by emphasizing the added value of a qualitative methodology in VR user evaluation studies and advocate for the use of mixed method approaches to better capture user perspectives.