A Three-Model Deep Learning Framework for ASL Recognition: Integrating CNN, Skeletal Features, and Multi-Modal Fusion

American Sign Language (ASL) recognition is an important way of enhancing deaf and hard-of-hearing individu-als' access to communication. In this present paper, we propose a multi-modal deep learning model with the incorporation of image-based, skeletal-based, and hybrid recognition methods in the attempt to foster enhanced ASL classification accuracy. Our method takes real-time video input and utilizes Convolutional Neural Networks (CNNs) for image-based recognition as well as Multi-Layer Perceptrons (MLPs) for skeletal feature extraction. MediaPipe Hands is used to identify 21 hand landmarks, which are further used as input for skeletal-based classification. The dataset, which is gathered from Kaggle, comprises 78,000 images of 26 ASL alphabets (A-Z), and each model is trained individually to achieve the best performance. An ensemble final model uses the most confident classification out of the three approaches to achieve maximum accuracy and reliability. The system is real-time with confidence-based filtering and TTS support for better usability. The experimental results show that the multi-modal fusion approach outperforms single-modality models with high accuracy even in the case of diverse lighting conditions and hand orientations. Future directions involve using the model for sentence-level ASL recognition with LSTMs and deployment with optimizations on mobile and edge devices using TensorFlow Lite.