Indonesian Cued Speech Transliterate System Using Convolutional Neural Network MobileNet

Individuals with hearing loss use Cued Speech as a system for effective communication. Automatic Cued Speech Recognition has the potential to greatly facilitate communication between individuals with hearing loss and others. However, several challenges exist in automatic Cued Speech recognition, including improving hand position detection accuracy, synchronizing hand movements with lip shapes, and expanding available datasets. Currently, researchers are limited to French and English Cued Speech datasets, which introduces additional complexities for this study. The aim of this research is to develop a Cued Speech dataset in Indonesian and apply Convolutional Neural Networks (CNN) for Cued Speech recognition. This research introduces a video dataset of hand gestures corresponding to Indonesian Cued Speech, which is crucial for the automatic recognition of Indonesian sign language and serves as a valuable resource for the deaf community. This dataset is also valuable to researchers in the field of sign language and hand gesture recognition. The model was validated using Convolutional Neural Network (CNN) with TensorFlow, using MobileNet V2 architecture. The main contribution of this research is the development of the first dataset for Indonesian Sign Language, as no dataset has ever been tested with CNN with MobileNet V2 architecture. This dataset consists of 7,800 images in. jpg format, with 80% allocated for training, 10% for validation, and 10% for testing. The implementation of CNN with MobileNet V2 architecture achieved an accuracy rate of 95.90%. Having successfully detected individual letters of the alphabet, future research is expected to extend this work to full-word recognition.