Low-Resource Speech Recognition by Fine-Tuning Whisper with Optuna-LoRA

In low-resource speech recognition, the performance of the Whisper model is often limited by the size of the available training data. To address this challenge, this paper proposes a training optimization method for the Whisper model that integrates Low-Rank Adaptation (LoRA) with the Optuna hyperparameter optimization framework. This combined approach enables efficient fine-tuning and enhances model performance. A dual-metric early stopping strategy, based on validation loss and relative word error rate improvement, is introduced to ensure robust convergence during training. Experimental data were collected from three low-resource languages in Xinjiang, China: Uyghur, Kazakh, and Kyrgyz. Compared to baseline LoRA fine-tuning, the proposed optimization method reduces WER by 20.98%, 6.46%, and 8.72%, respectively, across the three languages. The dual-metric early stopping strategy effectively shortens optimization time while preventing overfitting. Overall, these results demonstrate that the proposed method significantly reduces both WER and computational costs, making it highly effective for low-resource speech recognition tasks.