Hyper-adapter for Parameter-Efficient Multilingual ASR Adaptation

This work proposes a new parameter-efficient adaptation approach for multilingual ASR based on the hyper-network. Existing multilingual ASR adaptation methods apply either one residual adapter for all the languages, or language dependent adapters for each individual language. The residual adapter cannot compete the full finetuning in terms of WER, because it is agnostic to language information. Whereas the language dependent adapters introduce high parameter overhead without a parameter sharing strategy. In contrast, we leverage a hyper-network to generate the weights for the adapters across different languages. To achieve the best parameter sharing strategy that scales with a large number of languages, we propose multi-level conditioning vector fusion, orthogonal regularization to improve the hyper-network output diversity, and language loss weighting during the model training. The proposed approach demonstrates comparable or better WER and better parameter efficiency compared to previous multilingual ASR adaptation approaches on commonly used multilingual ASR benchmarks.