Intelligent Sign Language Interpretation System Using Multi-Modal Deep Learning Architectures

Sign language users face persistent communication barriers with non-signers, and human interpreters are costly and often unavailable. This paper presents a real-time system for recognizing American Sign Language (ASL) fingerspelling from a standard webcam. The proposed multi-modal architecture combines three complementary models: a ResNet based convolutional neural network (CNN) that extracts hand-shape features from image frames, a graph neural network (GNN) that models relationships between 21 MediaPipe hand landmarks, and a Vision Transformer (ViT) that applies self-attention over image patches. Their predictions are combined using weighted averaging to improve robustness to changes in lighting and background. The models were trained and evaluated on the ASL Alphabet dataset, containing approximately 87,000 images across 29 classes (letters A–Z plus space, delete, and nothing), using an 80/10/10 train/validation/test split. On the test set, the CNN, ViT, and GNN achieved accuracies of 99.83%, 93.96%, and 83.0%, respectively, while the ensemble achieved an accuracy above 95%. The system is deployed as a Flask and SocketIO web application that processes 640×480 webcam input at approximately 30 frames per second. It also supports progressive word construction, autocomplete, and sentence formation through SPACE and DELETE gestures. The current system recognizes static fingerspelled letters only. Extension to dynamic signs, word and sentence-level recognition, and facial and body cues is left for future work.