AI-Based Indian Sign Language Recognition, Real-Time Speech Generation Using Deep Learning and Computer Vision

Communication between people who use Indian Sign Language (ISL) and people who do not understand it can be challenging. This project presents a camera-based ISL recognition system that converts hand gestures into letters and words, supports rule-based sentence formation, and converts selected sentences into speech. OpenCV processes camera frames, while MediaPipe Hands extracts hand landmarks. The letter-recognition module uses 21 landmarks with X, Y, and Z coordinates, producing 63 features that are classified using a Random Forest model. The dataset contains 30,884 samples, and the reported accuracy is approximately 99.97%. The word-recognition module extracts 162 features per frame and processes 16-frame sequences using a TensorFlow/Keras architecture with Conv1D, Multi-Head Attention, and Transformer-based components. It recognizes 23 predefined words and achieves 87.67% accuracy. Recognized words are stored and passed to a rule-based sentence-formation module, where the user selects a sentence for Text-to-Speech using pyttsx3. An integrated Speech-to-Text module converts spoken input into text, supporting communication in the reverse direction. A Tkinter application provides learning and recognition modes for letters and words, sentence recognition, and Speech-to-Text.