The State of Replication at IEEE ISMAR and IEEE VR: A Scoping Literature Review (2010 – 2024) and Online Survey

A replication attempts to confirm outcomes of earlier research, which is critical in validating and generalizing scientific findings. Yet, its prevalence and practices remain underexplored in Augmented and Virtual Reality (AR/VR) research. To address this, we present a scoping literature review of replication studies within IEEE ISMAR and IEEE VR, spanning 15 years from 2010 to 2024. Our analysis revealed that replication in AR/VR research is rare. Of 2167 total papers reviewed, less than 4 % were identified as replication studies. Of these, conceptual replications were the predominant type (57 %). Most of those were studies in VR (67 %) and within-subject designs (66 %). Complementing the literature survey results, we conducted an online survey with 61 participants about their experiences with replication studies and found that 39 % of them had conducted a replication study. However, limited resources and external motivation hamper the execution of replication studies among these AR/VR researchers. Combining the findings from our literature review and online survey, we discuss the current state of replication research and the factors contributing to its infrequency. We provide recommendations to improve AR/VR research replication practices, focusing on research culture and reporting, and discuss ongoing challenges.