AI Gesture-Based Game Assistant with Real-Time Hand Tracking

Abstract Traditional gaming interfaces rely on physical controllers or keyboards, which can be restrictive and less accessible to certainusers. This paper presents an AI Gesture-Based Game Assistant with Real-Time Hand Tracking, a vision-based system that enables touchless game control through hand gestures and integrates an AI coaching assistant that provides contextual onscreen guidance during gameplay. Using a standard webcam, the system captures real-time video processed through OpenCV and MediaPipe for hand landmark extraction, obtaining 21 precise 3D landmarks per frame at 30+ FPS. A rule-based gesture recognition engine classifies five distinct gestures—Fist (Stop), Open Palm (Start), Thumb Up (Jump), Index Point (Shoot), and Swipe (Move)—achieving per-gesture accuracies of 85–96% with end to-end hand-movement-to-keypress latency under 50 ms. Recognized gestures are mapped to game-control events via PyAutoGUI/Pynput, enabling seamless interaction with a Pygame-based 2D game. Landmark smoothing through a sliding window deque and a frame-based debounce cooldown significantly reduce jitter and false triggers. Two of the four planned objectives have been fully implemented and validated; the remaining objectives—an AI coaching layer and an integrated real-time overlay—are scheduled for Phase 2. Index Terms—Gesture Recognition, Real-Time Hand Tracking, MediaPipe, Human-Computer Interaction, Touchless Gaming, OpenCV, Game Accessibility