Enhancing Multilingual ASR for Unseen Languages via Language Embedding Modeling

Multilingual Automatic Speech Recognition (ASR) aims to recognize and transcribe speech from multiple languages within a single system. By leveraging a vast amount of data and incorporating language tokens as prefixes to guide the recognition process, Whisper is one of the most advanced multilingual ASR models. However, despite its success, Whisper struggles with unseen languages, which are not included in its pre-training. Motivated by the observation that many languages share linguistic characteristics, we propose a method that exploits these relationships to improve ASR performance of Whisper in unseen languages. With Whisper’s predicted language probabilities as weight, the approach weighted sums the embeddings of language tokens . Based on this method, we also develop a predictor-based approach that supports our assumption about the weighted sum method. Our proposed methods demonstrate substantial improvements that outperform the baseline approaches, providing an effective solution for addressing unseen languages in the multilingual ASR task.