A Multimodal Approach for Targeting Error Detection in Virtual Reality Using Implicit User Behavior

Although the point-and-select interaction method has been shown to lead to user and system-initiated errors, it is still prevalent in VR scenarios.Current solutions to facilitate selection interactions exist, however they do not address the challenges caused by targeting inaccuracy.To reduce the effort required to target objects, we developed a model that quickly detected targeting errors after they occurred.The model used implicit multimodal user behavioral data to identify possible targeting outcomes.Using a dataset composed of 23 participants engaged in VR targeting tasks, we then trained a deep learning model to differentiate between correct and incorrect targeting events within 0.5 seconds of a selection, resulting in an AUC-ROC of 0.9.The utility of this model was then evaluated in a user study with 25 participants that identified that participants recovered from more errors and faster when assisted by the model.These results advance our understanding of targeting errors in VR and facilitate the design of future intelligent error-aware systems.