Adapting Large-Scale Pre-trained Models for Unified Dialect Speech Recognition Model

Recent advancements in deep learning techniques utilizing large-scale data, such as self-supervised learning, have significantly improved the accuracy of speech and language processing technologies for major world languages. However, for dialects with limited transcription resources, technologies like automatic speech recognition and search have yet to be realized at a practical level. This issue is particularly pronounced in Japanese dialects, which are classified into dozens of different and mixed dialects, and remains unresolved. In this study, we focus on two large-scale pre-trained models that have demonstrated top-tier performance in recent automatic speech recognition system research, and present examples of unified automatic speech recognition systems adapted for Japanese dialects, as well as the potential applications of the content detection task — query-by-example spoken term detection. Both compared models are trained on thousands or more hours of multilingual speech, with one being an automatic speech recognition model based on self-supervised learning and the other (Whisper) a model based on multi-task learning, including machine translation. Experiments on automatic speech recognition models are conducted using several tens of hours of adaptation data for both standard Japanese and Japanese dialects, which have distinct characteristics depending on the region. The result shows that the dialect-independent automatic speech recognition model based on the self-supervised learning pre-trained model and 3-step adaptation strategy achieves the best accuracy with a character error rate of 29.2%, suggesting that it is important to consider regional identity due to the diversity and limited resources of Japanese dialects.