Hybrid Deep Learning and Fuzzy Matching for Real-Time Bidirectional Arabic Sign Language Translation: Toward Inclusive Communication Technologies

Technological advances and AI tools can help address the challenges faced by deaf-mute individuals in different areas of social interaction. Existing tools mainly focus on one-way translation are limited by small vocabulary datasets, demand significant computational power without real-time experiments. To overcome these limitations, this study was conducted to design a bidirectional real-time translation mobile application between ArSL and Arabic text, aiming to enhance communication and learning opportunities for deaf individuals. The proposed system consists of two primary modules: sign to word and word-to-sign. The first module employs selected transfer learning models to translate Arabic sign images into text, while the second module integrates a fuzzy string-matching model to convert Arabic text into sign images. The system was developed using six convolutional neural network (CNN)-based deep learning models: AlexNet, ResNet152V2, YOLOv8n, Swin Transformer, InceptionV3, and Xception. Additionally, the study utilized ArSL dataset and the Arabic data dictionary, enhancing their diversity, accuracy, and completeness to improve adaptability. Experimental evaluations were conducted to assess the system's performance in terms of accuracy and processing speed. The results demonstrated exceptional accuracy rates across all CNN models, with YOLOv8n-cls achieving the highest accuracy at 99.9%, followed by Xception, Swin Transformer, and AlexNet at 99.0%, and InceptionV3 and ResNet152V2 at 98.0%. These results are close due to the inherent characteristics of the dataset, as well as the shared principles among utilized algorithms such as preprocessing, augmenting, cross validation and hyper-tuning methods. Notably, in terms of execution time, InceptionV3, AlexNet, and YOLOv8n models demonstrated their efficiency in recognizing each individual sign image, with times of 13 ms, 16 ms, and 67 ms, respectively, showcasing their suitability for real-time applications. Among them, YOLOv8n stands out, particularly in terms of processing speed and environmental adaptability. The experimental results demonstrate significant improvements in translation accuracy, outperforming baseline models. The findings confirm the feasibility of developing a robust and efficient real-time bidirectional ArSL translation system through the integration of deep learning models and fuzzy string-matching techniques. The proposed application has significant potential to bridge the communication gap between the deaf and hearing communities.