Probabilistic In-Plane Detection for Mid-Air Virtual Surface Interactions

In virtual and augmented reality, certain types of mid-air interaction, such as planar drawing or dragging interface elements, are most sensibly performed on a 2D surface. When this surface must be located in mid-air and without any form of passive haptic feedback, it can be very difficult to accurately determine the start and end of user input due to various sources of uncertainty. The conventional approach is to apply a simple depth threshold on the interaction plane and associate entry within this threshold as the start event and exit from this threshold as the end event. However, users struggle to maintain planar movement without any physical support, and unintended departures from the threshold are common, resulting in a frustrating user experience. In this work, we explore an approach that handles the exit events probabilistically so that the user interface can gracefully switch between operating modes, and avoid frustrating interruptions of user interactions. Our method is simple, interpretable, and can be easily deployed on commercial VR hardware. In a user study, we demonstrate that it reduces the number of unintended exits by a factor of 16 in a drag-and-drop task and to almost zero in slider and sketching tasks, compared to simple depth-based thresholding. Participants also found our method more usable and less demanding.