Indian Sign Language to Text/Speech Translation

Abstract: This work showcases the design of a real-time Indian Sign Language (ISL) to text and speech translation system for enhancing communication for those with hearing and speech disability. The system uses MediaPipe to obtain 3D hand landmark coordinates that are saved in a CSV dataset. A Feed-Forward Neural Network, namely a Multi-Layer Perceptron (MLP) in TensorFlow/Keras, is trained to identify 26 alphabetic symbols (A–Z) from 126 input features that describe the x, y, z coordinates of 21 hand landmarks per hand. Two hidden layers with dropout are used to avoid overfitting and a test accuracy of around 96% is achieved. The model is converted to TensorFlow Lite after training for integration into mobile environments in a lightweight manner. A Flask-based server exposed through Cloudflare tunnels accepts real-time landmark information from the mobile app, does inference on the TFLite model, and returns the predicted result. The app dynamically shows recognized letters, constructs words and sentences, and reads them out in speech using a Text-to-Speech (TTS) engine. Also, Swaram API integration supports multilingual audio output, thus rendering the system universally accessible to varied linguistic users. This end-to-end solution is a scalable, efficient, and accessible method of ISL recognition and speech translation