HanGE: Enhancing Hand-Object Interaction in Virtual Reality with Semi-Transparent and Interpenetrable Hands Using Multiscale 3D Deep Convolutional Neural Networks

To perform the proposed title study, we chose the hand gesture applications. This well-known Virtual Reality (VR) application is more familiar due to its natural alignment with human-computer real-time interaction, providing a more immersive user experience. However, hand-based object manipulation in VR involves considerable limitations, such as obstructions. The unclear vision of objects makes the manipulation more difficult. These limitations sometimes need to be clarified when identifying objects in a VR environment. To address these challenges, the study aims to create semi-transparent and interpenetrable hands to improve user experience with the knowledge of deep learning. These hands help avoid physical constraints, improve precision and provide control during object manipulation tasks. We proposed the (Hand Gesture Intelligent & Interactive Recognition Algorithm) HanGE algorithm, designed by an advanced Multiscale 3D-Deep Convolutional Neural Network (M3D-DCNN) to design these novel VR hands. The study’s novelty lies in developing semi-transparent and interpenetrable hands using M3D-DCNN to improve hand–object interaction in VR. To evaluate the performance of HanGE, we conduct the experiments in a VR environment; participants are asked to perform various object manipulation tasks with both semi-transparent and interpenetrable hand configurations by using the input as a Leap Motion Controller (LMC). The proposed approach is assessed in terms of accuracy and task completion speed. Therefore, the additional feedback was collected from participants based on their experience. The evaluation results show significant improvement in precision and efficiency, highlighting the proposed ability of HanGE to improve VR interaction.