Investigating common hands-free interactions in supporting seamless cross-reality multitasking

Mixed reality technologies enable simultaneous performance of both virtual and real-world tasks. However, interference arises when both tasks require manual control, especially when the real-world task requires both hands, leaving little or no manual capacity for interacting with the virtual system. One promising approach is employing hands-free interactions for virtual tasks, freeing users' hands for physical activities. Yet, their effectiveness during concurrent real-world tasks remains underexplored. Addressing this gap, this study investigated four common hands-free interactions, Gaze, Head, Body and Face against the Controller baseline in supporting cross-reality multitasking. Two real-world task difficulty levels were implemented to simulate scenarios involving varying visual load, task complexity, and cognitive load. Multitasking and task difficulty showed small-to-large effect sizes. Generally, all hands-free interactions showed an advantage over Controller in real-world task, but not in virtual task under hard condition. Head showed better overall effectiveness than other hands-free interactions. Compared to Head, Body was associated with greater physical effort, discomfort, cybersickness, and impaired real-world spatial orientation, hand dexterity and visual stability. Nevertheless, Body is more immersive. Gaze and Face had high control acuity and required minimal movement. However, they were prone to accidental activations, marked eye discomfort, and visual attention conflict between virtual and real-world tasks, highlighting limitations for eye-based interaction during cross-reality multitasking with continuous VR operations. Overall, as a first exploration, our results suggest that hands-free interactions are feasible for cross-reality multitasking, while future work should explore multiple multimodal combinations and automation to further improve usability and performance.