Voice-augmented extended reality digital twin instruction: the case of the Rubik’s Cube

Abstract Extended Reality (XR) and Digital Twins (DTs) present promising opportunities for immersive, state-aware procedural learning, making them well-suited for step-by-step instructional tasks. Their ability to simulate real-world contexts aligns closely with experiential learning principles, which emphasize active engagement, reflection, and learning by doing in authentic environments. Building on this potential, our prior work introduced a DT-driven XR framework for learning procedural Rubik’s Cube notations, comparing three instructional modalities— a baseline visual static information condition with physical cube manipulation (M1), in-situ visual overlays with physical cube manipulation (M2), and in-situ visual overlays with virtual-then-physical cube manipulation (M3). However, the potential of multimodal guidance remained underexplored. This study extends the previous work by introducing and evaluating a fourth instructional modality, namely voice-augmented visual guidance (M4), which synchronizes auditory narration with in-situ visual overlays during physical cube manipulation. We assessed all four modalities across two post-training phases using task accuracy, completion time, and intra-action pause duration, alongside global measures of usability, acceptance, and cognitive effort. In the evaluated setting, M4 did not yield statistically significant differences relative to the other modalities (M1–M3), but achieved comparable performance and user-experience outcomes without additional cognitive or usability costs. These results delineate boundary conditions for multimodal instruction in this context and suggest that voice augmentation is best viewed as a complementary design element rather than an inherently performance-enhancing mechanism.