A Multi‐User Virtual Reality Game Based on Gesture Recognition From RGB Information

ABSTRACT In the ever‐evolving world of gaming, controller‐based input is quickly becoming obsolete. Various applications, including virtual reality (VR) and augmented reality (AR) games, have used motion‐ and gesture‐controlled video game consoles. The current state of the art relies on depth images of the hand that do not utilize color information from the RGB spectrum. In this work, we focus on the development of an interactive VR game that utilizes hand pose recognition from the RGB domain to increase user experience, but also simplify the functionality of the game. To address this challenge, a 3D multi‐user VR game themed around a “tennis match” was developed using the Unity engine. We also investigate whether we can estimate the coordinates of colored objects connected to the hand movement of the players and track human gestures to navigate through the game functions in real time using an RGB camera. Statistical analysis showed that the user experience increased concerning engagement and satisfaction using a more natural form of control that allows players to focus on the excitement of the game without worrying about button presses or joystick movements. Such hand pose recognition systems can be implemented to replace the traditional controller‐based entry systems used today.