LSTM-Based Hand Gesture Recognition for Indonesian Sign Language System (SIBI) on Affix, Alphabet, Number, and Word

Sign language plays a critical role in enabling communication for the Deaf and hard-of-hearing community in Indonesia, yet there remains a significant gap in technological support for recognizing the official Indonesian sign language, Sistem Isyarat Bahasa Indonesia (SIBI). This study presents a deep learning-based hand gesture recognition system for SIBI, focusing on four primary gesture categories: affix, alphabet, number, and word. A large and diverse dataset of 21,351 videos was collected, covering 18 affix, 26 alphabet, 35 number, and 29 word classes. Hand keypoints were extracted using MediaPipe Holistic, and a bidirectional long short-term memory (BiLSTM) model was trained using 5-fold stratified cross-validation. The model achieved high recognition performance in the alphabet, number, and word categories, with mean test accuracies of 93.94%, 91.48%, and 92.41%, respectively, and slightly lower performance in the affix category at 68.17%. The affix category posed particular challenges due to subtle hand shape differences and high variability between signers, while the alphabet category consistently showed the highest accuracy due to its distinct and static handshapes. Evaluation metrics, including precision, recall, F1-score, and confusion matrix analysis, provided further insights into model strengths and limitations. Overall, the study demonstrates the effectiveness of LSTM models for sequential hand gesture recognition in SIBI and highlights areas for future improvement, such as handling non-manual features and improving generalization across signers.