Domain adaptation using non-parallel target domain corpus for self-supervised learning-based automatic speech recognition

The recognition accuracy of conventional automatic speech recognition (ASR) systems depends heavily on the amount of speech and associated transcription data available in the target domain for model training. However, preparing parallel speech and text data each time a model is trained for a new domain is costly and time-consuming. To solve this problem, we propose a method of domain adaptation that does not require the use of a large amount of parallel target domain training data, as most of the data used for model training is not from the target domain. Instead, only target domain speech is used for model training, along with non-target domain speech and its parallel text data, i.e., the domains and contents of the two types of training data do not correspond to one another. Collecting this type of training data is relatively inexpensive. Domain adaptation is performed in two steps: (1) A pre-trained wav2vec 2.0 model is further pre-trained using a large amount of target domain speech data and is then fine-tuned using a large amount of non-target domain speech and its transcriptions. (2) The density ratio approach (DRA) is applied during inference to a language model (LM) trained using target domain text unrelated to, and independently from, the wav2vec 2.0 training. Experimental evaluation illustrated that the proposed domain adaptation obtained character error rate (CER) 10.4 pts lower than baseline with wav2vec 2.0 and 3.9 pts with XLS-R under the situation that the parallel target domain data is unavailable against the target domain test set, achieving 34.4% and 16.2% reductions in relative CER. • For the self-supervised learning (SSL) model-based automatic speech recognition (ASR), we developed a method to adapt the ASR model to speech in the domain in which there are in-domain speech and text data, but there are no parallel speech and text data. • The acoustical adaptation method consists of an additional pre-training of an SSL model using target domain speech data and a fine-tuning of the entire ASR model using source domain (out-of-domain) speech data. • We also combined the ASR model with a language model (LM) using shallow fusion (SF) or density ratio approach (DRA). • The experimental results showed that our approach was effective for domain adaptation. • The acoustical and linguistic adaptations worked complementarily.