From words to proverbs: Evaluating LLMs’ linguistic and cultural competence in Saudi dialects with Absher

As large language models (LLMs) become increasingly central to Arabic NLP applications, their effectiveness in linguistically diverse settings, particularly regions with rich dialectal variation such as Saudi Arabia, remains underexplored. Existing evaluation paradigms tend to prioritize high-resource languages or Modern Standard Arabic (MSA), overlooking regional linguistic and cultural specificities. This leads to performance limitations and cultural biases in real-world deployments. To address this gap, we introduce Absher , the first comprehensive and fine-grained benchmark designed to assess the understanding of LLMs regarding Saudi dialects and their embedded cultural nuances. Absher consists of over 18,000 multiple choice questions derived from a curated dataset of dialectal words, phrases, and proverbs sourced from five major Saudi regions. The benchmark spans six task categories: Meaning, True/False, Fill-in-the-Blank, Contextual Usage, Cultural Interpretation, and Location Recognition, enabling multifaceted evaluation across both linguistic and cultural dimensions. We perform zero-shot evaluations on six state-of-the-art open LLMs: ALLaM, LLaMA, Jais, Mistral, Qwen, and AceGPT. Our results reveal substantial performance variability across dialects and question types. Qwen achieved the highest overall accuracy, excelling in word-level questions (63%), while ALLaM outperformed others in the interpretation of proverbs (48% accuracy). All models struggled with content from underrepresented dialects, particularly Southern and Eastern variants, and with context-free True/False questions, highlighting weaknesses in dialect grounding and binary reasoning. These findings demonstrate the need for dialect-aware training and culturally aligned evaluation. We position Absher as a critical step toward more equitable and effective LLMs development for real-world Arabic applications.