Efficient Cross-Lingual Transfer for Language Models

Pre-trained language models (LLMs) dominate NLP research, especially for downstream tasks, but primarily focus on English, leaving other languages at a disadvantage. Multilingual LLMs show promise but often underperform compared to their monolingual counterparts. This paper explores techniques for adapting fine-tuned English LLMs, specifically those utilizing in-context learning (ICL), to other languages. We evaluate the WECHSEL language adaptation technique, applying it to the large variant of GPT-2 for French and German. Our contributions include releasing new model checkpoints, assessing robustness and performance on downstream tasks, and introducing “targeted distillation” via vessel adapters (PHODIVA). Despite promising techniques, our baseline translation method surprisingly rivals the adapted models, suggesting directions for future research.