Regional Dynamic Sign Language Recognition using Multimodal Conformer Architecture

This paper proposes a multimodal Conformer architecture for dynamic sign language recognition in Kannada Sign Language (KSL). The model incorporates visual features extracted by EfficientNet-B0 together with 3D hand key points obtained from MediaPipe. The paper further proposes an Adaptive Confidence Correction (ACC) strategy, supported by k-NN classification, when the confidence scores for the hand key points of the signs are low and inconsistent. Finally, the dataset included 1,180 video samples covering 11 dynamic signs. Our tests demonstrate that the presented method achieves a state-of-the-art accuracy of 98.63% with a low inference time of 58ms while outperforming baselines including CNN-LSTM, Transformer, I3D, and SlowFast. Cross-validation tests and statistical analyses further support the robustness of the presented work. This work makes the following key contributions (1) the development of a newly collected in-house dataset Kannada Sign Language (KSL) dataset addressing the data scarcity of underrepresented regional sign languages, (2) the first adaptation of the Conformer architecture for sign language recognition, validated through ablation and cross-validation studies, and (3) a deployable, low-latency framework designed for mobile-edge integration and privacy-aware deployment. The dataset will be released upon acceptance and with a valid research request. By addressing accessibility challenges in human–computer interaction and offering a reproducible benchmark for regional sign-language technologies, this work supports future advances in cross-lingual generalization and low-resource optimization.