Exploring prompting for dialectical machine translation: a focus on north Jordanian Arabic

Dialectal variations are common across many languages, and dialectical machine translation to the standard form of the language or other languages is crucial for effective communication with speakers of these dialects. Prompting Large Language Models (LLMs) for Machine Translation (MT) has gained popularity. However, its efficacy for dialectical MT, particularly in comparison to fine-tuning, remains underexplored, especially for regional dialects that lack parallel training and evaluation data. This study presents a new parallel dataset between Modern Standard Arabic and the Irbid dialect, the largest city in northern Jordan, specifically within the travel domain. This dataset, an extension of the MADAR multi-dialect corpus , comprises 12,000 entries translated by native speakers of the Irbid dialect. We also describe the guidelines and evaluation process employed to collect this dataset and present several analyses within this article. Additionally, we investigate the effectiveness of prompting LLMs, particularly GPT-4o-mini, in performing MT under zero-shot and few-shot learning settings. We compare these methods to fine-tuning approaches. This includes the use of dialect-tolerant prompts and constraints. We compare these methods to fine-tuning approaches. Results indicate that prompting, particularly few-shot learning with an optimal number of exemplars, consistently outperforms fine-tuning in our tests. Utilizing several versions of T5 and mBART50 for fine-tuning, we compared their performance with that of GPT-4o-mini, which was employed for prompting. The comparative analysis reveals a notable improvement margin, with Bilingual Evaluation Understudy (BLEU), Crosslingual Optimized Metric for Evaluation of Translation (COMET), and Recall-Oriented Understudy for Gisting Evaluation–Longest Common Subsequence (ROUGE-L) scores surpassing those of the best fine-tuned model by margins of 11.89, 0.2476, and 1.18, respectively. These findings underscore the potential of Few-shot Prompting (FSP) in effectively addressing dialectical MT challenges.