Improving Dialect Robustness in Large Language Models via LoRA and Mixture-of-Experts

Despite the success of large language models (LLMs) in a wide range of applications, it has been shown that their performance varies across English dialects. Differences among English dialects are reflected in vocabulary, syntax, and writing style, and can adversely affect model performance. Several studies evaluate the dialect robustness of LLMs, yet research on<br/>enhancing their robustness to dialectal variation remains limited.<br/><br/>In this paper, we propose two parameter efficient frameworks for improving dialectal<br/>robustness in LLMs: <i>DialectFusion</i> where we train separate LoRA layers for each dialect and<br/>apply different LoRA merging methods, and <i>DialectMoE</i> which is built on top of Mixture of Experts LoRA and introduces multiple LoRA based experts to the feed-forward layer to internally model the dialectal dependencies. Our<br/>comprehensive analysis on five open-source LLMs for sentiment and sarcasm tasks in zero and few-shot settings shows that our proposed approaches enhance the dialect robustness of LLMs and outperforms instruct and LoRA fine tuning based approaches.