Enhancing YOLOv3 with RNN Models: application to American Sign Language recognition for deaf individuals

As a natural language, sign language is a rich and expressive form of communication with properties similar to spoken languages. In this paper, we explore the application of transfer learning techniques to develop an intelligent system for recognizing American Sign Language. Instead of building a new deep convolutional neural network from scratch, we leveraged pre-trained models to capitalize on their advantages and reduce experimentation time. Specifically, we focused on YOLOv3 combined with advanced recurrent layers, including LSTM, GRU, and Bidirectional LSTM. The results were highly promising: YOLOv3 + Bidirectional LSTM achieved the highest accuracy, averaging over 94%, followed by YOLOv3 + LSTM at ~93%, and YOLOv3 + GRU at ~91%. These findings highlight the effectiveness of combining YOLOv3 with recurrent architectures for sign language recognition, demonstrating robust precision, recall, and F1 scores across models, with Bidirectional LSTM standing out as the top-performing configuration.