Low-Resource Language Adaptation Using Two-Stage Regularization for Multilingual ASR

A significant portion of the global population speaks multiple languages, including many low-resource languages on which current multilingual ASR models perform poorly. To improve ASR performance on these languages, models are typically fine-tuned on low-resource data with only a small subset of parameters adjusted to prevent overfitting. Traditionally, separate models are fine-tuned for each target language. During inference, a specific model is selected based on the language to be transcribed. However, in applications such as smart home devices, where multilingual users may vary their language use unpredictably, it becomes challenging to pre-select the appropriate model when the language is unknown in advance. To address this, this paper introduces a two-stage regularization-based continual learning method that adapts a single model to transcribe both source and target languages while minimizing overfitting. In the first stage, strong regularization is applied to update all parameters, and in the second, regularization is relaxed to enhance learning capacity. By adapting Whisper, a state-of-the-art speech foundation model that supports MASR, to 10 h of data for each of 2 target languages from Common Voice, the results show that our method can reduce average word error rate in the source and target languages from 18.59% to 15.52%.