WhispAdapt: Multilingual dialect recognition in low-resource Dravidian languages

Dialect variation in pronunciation, vocabulary, grammar, and syntax poses a persistent challenge for multilingual speech systems. This challenge is especially noticeable in low-resource Dravidian languages. Existing models often struggle due to the lack of annotated dialectal corpora and poor adaptability to regional speech patterns. To tackle this gap, we propose WhispAdapt, a lightweight and efficient adaptation of the Whisper encoder built for dialect classification tasks. Our framework applies parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA) and its quantized variant (QLoRA), allowing the model to learn effectively even with fewer samples while keeping computation and memory demands low. We made a 65.9-hour dialectal speech dataset with twelve dialects from Tamil, Kannada, and Malayalam for evaluation. The experimental results demonstrate that WhispAdapt-LoRA achieves impressive results compared to the other variant-WhispAdapt-QLoRA. It achieved macro F1-scores of 93% for Kannada, 97% for Malayalam, and 90% for Tamil—surpassing benchmarks such as Wav2Vec2, HuBERT, and WavLM by as much as 25%. Moreover, WhispAdapt also generalizes effectively to unseen accents on the Accent_DB benchmark. Overall, this study offers a scalable step toward improving dialect modeling for other underrepresented languages.