Development of a hybrid CNN-RNN model for enhanced recognition of dynamic gestures in Kazakh Sign Language

With around 1 % of the population of the Republic of Kazakhstan being affected by hearing disabilities, Kazakh Sign Language holds great importance as a means of communication between citizens of the state. The limitations of tools for Kazakh Sign Language (KSL) create significant challenges for people with hearing impairments in education, employment, and daily interactions. This research addresses these challenges through the development of an automated recognition system for Kazakh Sign Language gestures, aiming to enhance accessibility and inclusivity of communication using artificial intelligence. The approach employs advanced machine learning techniques, including Convolutional Neural Networks (CNNs) for recognizing spatial gesture patterns and Recurrent Neural Networks (RNNs) for processing temporal sequences. By combining these methods, the system recognizes both hand gestures and facial expressions, providing a dual-stream model that surpasses single-stream gesture recognition systems focused solely on hand movements. A dedicated dataset was created using Mediapipe Holistic, an open-source tool that identifies 543 landmarks across hands, faces, and poses, effectively capturing the multifaceted nature of sign language. The findings showed that the hybrid model significantly outperformed standalone CNN and RNN models, achieving up to 96 % accuracy. This demonstrates that integrating facial expressions with hand gestures greatly enhances the precision of sign language recognition. This system holds immense potential to improve inclusivity and accessibility in various settings across the Republic of Kazakhstan by facilitating communication for hearing-impaired individuals, paving the way for expanded research and application in other sign languages