A Parameter-Efficient Multi-Step Fine-Tuning of Multilingual and Multi-Task Learning Model for Japanese Dialect Speech Recognition

This paper addresses the challenge of developing a unified spoken language model for Japanese dialects. Due to the limited availability of such speech resources, self-supervised learning (SSL) methods show promise. However, the diversity of Japanese dialects also requires multilingual approaches to language modeling. One of the few related works improved performance through multi-step fine-tuning, using standard Japanese speech and multitask learning that incorporates dialect identification. However, this method is computationally expensive and runs the risk of forgetting acquired knowledge. To address these challenges, we propose a large-scale multilingual SSL model-based multistage fine-tuning strategy using lightweight adapter modules per domain. Our method achieves a relative reduction in character error rate of 18% compared to simple full fine-tuning method, especially in the speaker adaptation scenario, while using only half the training parameters required for full fine-tuning.