ADVANCED TOUCHLESS HCI: A GESTURE RECOGNITION-BASED MULTILINGUAL VIRTUAL KEYBOARD FOR CHARACTER INPUT

Abstract Virtual keyboard-based non-touch character input systems present an advanced communication method between humans and computers, offering interaction in challenging environments like industrial settings. Extensive research has explored touch and touchless input methods, including hand gestures, aerial handwriting, sign language recognition, and finger alphabet systems. However, many systems require significant learning and complex processing for accurate character recognition. This reveals the need for more efficient, accessible, and low-overhead solutions in non-touch input technologies. This paper presents a virtual keyboard-based character input system that utilizes hand gesture detection to create a novel touchless human-computer interaction (HCI) interface. The study has two key components: a hand gesture recognition system and a character input method. The system leverages MediaPipe's pre-trained models to accurately detect human body keypoints, enabling mid-air typing through intuitive hand gestures. We calculated the angles and distances between various keypoints to extract the features for gesture recognition. OpenCV is used for data collection, and Pynput facilitates keyboard control. A CronoNet architecture-based model powers the system, translating hand gestures into precise keyboard inputs. The virtual keyboard supports seamless transitions between language layouts, including English and Bengali. It recognizes gestures for commands such as scrolling (up/down), swiping (left/right), thumbs up, and finger tapping for input. The system achieved an average accuracy of 96.54% in gesture recognition and 97.07% in character input, showcasing its superiority over state-of-the-art methods.