Domain Adaptation by Alternating Learning of Acoustic and Linguistic Information for Japanese Deaf and Hard-of-Hearing People

More than half of Japanese people with hearing impairments communicate using speech, however speech recognition systems trained using speech from individuals with normal hearing are unable to achieve sufficient recognition accuracy of speech from individuals with hearing impairments. Therefore, speech recognition systems adapted for individuals with hearing impairments are needed. In this study, we propose a learning method that retains both acoustic and linguistic information to achieve more accurate recognition of speech of the hearing-impaired. Our proposed method performs domain adaptation by alternately switching whether to train the Transformer encoder layer and decoder based on the input speech. By using this method, we can create a speech recognizer adapted to hearing-impaired speech acoustically, while preserving the linguistic information of a general training corpus, thereby improving recognition performance for hearing-impaired speech.