Advancing Visual Speech Recognition in Indonesian Language Through Cross-Modal Knowledge Distillation

Visual Speech Recognition (VSR) systems have shown significant advances worldwide, but research and resources remain sparse for low-resource languages like Indonesian. This study aims to address this gap by developing a VSR model specifically tailored for the Indonesian language, utilizing Knowledge Distillation (KD) to improve performance. Using the LUMINA dataset, we trained a baseline VSR model and a KD-enhanced VSR model, where the teacher is an Automatic Speech Recognition (ASR) model. Both models utilize the Conformer-Based architecture for encoding, with adjustments to accommodate the distinct characteristics of video input. Our proposed model resulted in an improvement of 2.51% in Word Error Rate (WER) compared to the baseline VSR model. Furthermore, the integration of an LM decoder further decreases the WER by 0.75%. This research highlights the potential of KD and LM integration to address the challenges inherent in visual information and improve the performance of VSR systems for the Indonesian language.