Developing a CNN and NLP Integrated System for Translating American Sign Language Into Human Language

American Sign Language (ASL) is an important means of communication among deaf and speech-impaired people. The communication gap, however, exists between users of ASL and users of the spoken language. Thus, to fill this gap, we propose a system that would translate ASL into text and speech by utilizing advanced deep learning and NLP techniques. A high-resolution camera captures real-time video of an ASL gesture, and this video is then processed using MobileNetV3 to achieve key hand landmarks. These are very important for accurate gesture recognition as they provide essential data about the shape and movement of the hand. A CNN model is then separately trained on such extensive ASL datasets that can do precision recognition of particular gestures based on both spatial and temporal information analysis. These identified gestures are then propagated through an NLP model that involves generation of text and speech outputs which are both contextually accurate as well as grammatically correct. The scalable architecture of the system and also efficient ensures that the system in its nature adapts itself to different dialects of the ASL with minimal retraining. The system was evaluated on a rich ASL dataset, achieving high accuracy for gesture detection and translation, thereby showing its promise to bridge the gap between ASL users and the community at large.