RTSLIS: A Real-Time Sign Language Interpretation System Based on Vision Transformers

Sign language is an important form of communication for non-verbal, deaf and hard-of-hearing individuals, but it is still very disconnected from mainstream communication technology. Sign language can be difficult to communicate because it varies regionally and culturally across the globe. In complex situations, such as hospital emergencies, customer service, buyer-seller communication or criminal cases millions of deaf and hard-of-hearing people face major challenges. Our research objective is to develop sign language recognition systems where each sign directly assigns each cabalistic sentence. To address these challenges, we propose the development of a universal standard for sign language in emergency situations, ensuring consistent and effective communication across the globe. Our system employs state-of-the-art vision transformers for identifying sign language gestures and translating them into textual sentences with high accuracy. We have designed and implemented the system using the PyCharm environment, it carries out not only recognition but also real-time translation by showing the interpreted sentences separately in a window. We use a one-handed gesture approach for our dataset. We create a custom dataset containing about 5440 images for 20 classes. Our system handles both image and video input. We trained our dataset with different models such as YOLOv5(You Only Look Once Version 5), YOLOv8, YOLOv9, Faster R-CNN (Faster Region-based Convolutional Neural Network) and RetinaNet. Our method simplifies the translation process and provides accurate and efficient communication with remarkable accuracy. For all YOLO versions, the best accuracy is 99.64%, for Faster R-CNN the accuracy is 71.4%, and for RetinaNet the accuracy is 67.94%.