Multi-Modal ISL-Speech: A Real-Time Neural Architecture for Sign Language Translation to Natural Language

This research study presents a novel neural architecture for real-time Indian Sign Language (ISL) translation to regional languages, incorporating advanced optimization techniques to enhance both accuracy and computational efficiency. We introduce a hybrid attention mechanism combining spatial and temporal attention with quaternion-based neural networks (QNN) for improved gesture recognition. The architecture employs neural architecture search (NAS) with hardware-aware constraints to optimize model deployment on edge devices. Our approach integrates a novel adaptive pruning strategy that dynamically adjusts model complexity based on input complexity, significantly reducing latency while maintaining high accuracy. Experimental results demonstrate a 27% improvement in translation accuracy compared to existing methods, while achieving a 40% reduction in inference time. The system successfully translates ISL to five major Indian languages with an average accuracy of 94.2% and an end-to-end latency of 45ms on consumer-grade hardware.