Deep Neural Network-Based Alignment of Virtual Reality onto a Haptic Device for Visuo-Haptic Mixed Reality

Precise alignment between virtual reality (VR) and haptic interfaces is essential for delivering an immersive visuo-haptic mixed reality experience. Existing methods typically depend on markers, external trackers, or cameras, which can be intrusive and hinder usability. In addition, previous network-based approaches generally rely on image data for alignment. This paper introduces a deep neural network-based alignment method that eliminates the need for such external components. Unlike existing methods, our approach is designed based on coordinate transformation and leverages a network model for alignment. The proposed method utilizes the head-mounted display (HMD) position, fingertip position obtained via hand tracking, and the six-degrees-of-freedom (6-DOF) pose of a haptic device’s end-effector as inputs to a neural network model. A shared multi-layer perceptron and max pooling layer are employed to extract global feature vectors from the inputs, ensuring permutation invariance. The extracted feature vectors are then processed through fully connected layers to estimate the pose of the haptic device’s base. Experimental results show a mean positional error of 2.718 mm and the mean rotation error of 0.5330°, which equates to 1.3% relative to the haptic device’s maximum length. The proposed method demonstrates robustness against noise, demonstrating its applicability across various domains, including medical simulations, virtual prototyping, and interactive training environments.