Recognizing Indonesian words based on visual cues of lip movement using deep learning

Lipreading is one of the techniques that can enhance speech perception. However, there are still limited studies of lipreading research focusing on low-resource languages, such as Indonesian. In this study, we introduce an instrument designed to generate lipreading datasets using CC BY video data available on YouTube called Lipreading Information Resource Assembler-Generator (LIRA-Gen). Using this instrument, we present the first Indonesian language lipreading dataset (IDLRW) containing over 48,000 videos with 100-word categories spoken by various persons in natural conditions. Also, we developed a deep learning architecture consisting of an Advanced Residual Network (ARN) using ResNet-34 incorporated with a Channel Spatial Attention (CSA) module, improved sequence modelling by fusing Bi-Gru with Mamba (BGM), an integrated word decision module, and fine-tuned hyperparameter. Our measurement shows that it reaches an accuracy of 60.51% on the IDLRW dataset and outperforms state-of-the-art lipreading models from another dataset even without implementing an additional learning strategy. • Introduced a community tool to produce lipreading dataset. • Presented IDLRW, the first Indonesian lipreading dataset. • Proposed novel architecture with CSA and BGM for enhanced performance. • Metrics measurement: accuracy, precision, recall, f1-score, model parameters, FLOPS. • Achieved state-of-the-art in our IDLRW, outperforming other notable studies.