Real-time Gestures in Handheld Augmented Reality with Markerless Tracking for Multiplayer 3D Game

Augmented reality (AR) multiplayer games have surged since the launch of Pokémon Go and Ingress, which leverage AR markerless tracking. Invoking real hand gestures in handheld AR could enhance the experience in multiplayer 3D games. However, 3D games in handheld primarily use touchscreen interactions, limiting player freedom to explore the AR interactivity. This research aims to design a multiplayer AR game using real hand gestures with markerless tracking to enhance player interaction. Tower defense 3D game as a prototype to invoke the multiplayer. The objectives are to study real hand gestures and markerless tracking in handheld AR and implement the gesture into multiplayer AR games. ARCore facilitated markerless detection by scanning the real environment and gathering the necessary information to register AR content in AR extensions or AR cloud anchors. This data was stored in Google Cloud, generating an API that could reproduce the AR content. The real hand gestures have allowed the multiplayer to manipulate the AR content in a realistic way, enhancing the overall gameplay. By leveraging vision-based tracking mechanisms Manomotion gesture, the system ensured precise detection and interpretation of hand movements.