Enhancing Sign Language Interpretation with Multiheaded CNN, Hand Landmarks and Large Language Model(LLM)

Sign language is an important mode of communication for the deaf and mute community. Despite its importance, there is still a large communication gap between deaf community and the hearing world. We introduce a new system that converts sign language into text, using a novel multiheaded Convolutional Neural Network (CNN) that is trained on three different sets of images-raw images, uniquely segmented images, and hand landmarks information simultaneously to recognize sign language gestures accurately for different hand textures under disparate background conditions. Additionally, a Large Language Model (LLM) is incorporated to transform these recognized signs into concise and meaningful sentences. This combination ensures the text is both understandable and grammatically correct, thereby reducing the communication delay that exists in multiple Sign Language Recognition (SLR) systems. The model excels in capturing nuanced features and variations in sign gestures. Tests show that our novel approach is highly effective in real-time recognition, demonstrating its potential to improve communication for the Deaf community greatly.