Multimodal Feature Fusion with Hybrid CNN for Real-Time Sign Language Recognition in Human–Robot Interaction

Sign language recognition (SLR) is a pivotal task in enabling inclusive human–computer interaction and real-time surveillance applications. Hand gesture-based SLR represents a transformative advancement in assistive technology that enables communication for deaf and hard-of-hearing people. Automated systems for this language have wide applications in computer vision and human-computer interaction. There exist several sophisticated algorithms and machine learning (ML) models that translate hand gestures into comprehensible language. However, the real-time capabilities of these systems lack the desired precision. This study fosters a novel approach for hand gesture detection for enhanced accuracy and improved performance. The proposed approach integrates local binary pattern (LBP) features with a robust convolutional neural network (CNN). LBP features capture the fine details of hand gestures which are used to enhance the discriminative power of the CNN model. The robustness of the approach is validated through extensive experiments, demonstrating significant improvements in accuracy and reliability compared to existing methods. Integrating LBP features with the deep learning CNN model, the proposed approach achieves superior performance with a 99.9% accuracy. Unlike traditional SLR systems that depend only on deep learning models or handcrafted features, this work presents a hybrid multimodal feature fusion approach (combining LBP and CNN). This method not only achieves near-perfect accuracy but also ensures real-time performance, setting a new benchmark for SLR in surveillance and assistive robotics applications.