Real-time HGR using Handcrafted Descriptors and Late Fusion Transformers for Computer Interaction

In this paper we develop a vision-based Hand Gesture Recognition (HGR) system for real-time computer mouse control by leveraging alternate deep learning techniques, post-processing methods, and dataset refinements. The goal is to enhance classification accuracy, reliability, and system usability for real-time applications. By integrating Google’s Mediapipe Hands for hand tracking and incorporating dynamic gesture recognition techniques, the system extracts raw skeleton data for two main classifiers: a static hand pose MLP classifier for precise frame-by-frame classification and a dynamic gestures classifier using late fusion temporal transformers window prediction of dynamic gestures mapped to computer mouse macro actions. The system achieved an overall accuracy 85.86% and a Levenshtein accuracy of 73.86%, indicating an optimized, more efficient, and reliable HGR system that offers improved functionality and enhanced usability in practical applications.