ArGAN: Arabic Gender, Ability, and Nationality Dataset for Evaluating Biases in Large Language Models

Large language models (LLMs) are pretrained on substantial, unfiltered corpora, assembled from a variety of sources.This risks inheriting the deep-rooted biases that exist within them, both implicit and explicit.This is even more apparent in low-resource languages, where corpora may be prioritized by quantity over quality, potentially leading to more unchecked biases, particularly in low-resource languages, where all available data is leveraged solely to expand volume due to inherent scarcity.More specifically, we address the biases present in the Arabic language in both general-purpose and Arabic-specialized architectures in three dimensions of demographics: gender, ability, and nationality.We introduce ArGAN, a dataset for evaluating the fairness of these models across three demographic axes: gender, ability and nationality.Where we experiment with bias-revealing, template-based prompts and measure performance and bias using existing and evaluation metrics, and propose adaptations to others.