A test of meaning, form, and culture in Kurmanji: An evaluation of large language models’ performance

<ns3:p>Background Large Language Models (LLMs) are increasingly applied to low-resource languages. However, their performance in morphologically rich and culturally embedded languages such as Kurmanji remains underexplored. This study addresses how major LLMs process linguistic accuracy, semantic nuance, and cultural representation in Kurmanji. Methods Four state-of-the-art LLMs—ChatGPT-4o, Gemini, Claude, and Grok—were evaluated using a 40-item Kurmanji test set. Each prompt targeted specific aspects of language competence, including grammatical correctness, semantic interpretation, cultural appropriateness, and idiomatic usage. Responses were assessed by native-speaking annotators with a rubric focusing on meaning preservation, formal adequacy, and cultural fit. Quantitative scores were supplemented by qualitative analyses of error types and model-specific tendencies. Results The models demonstrated divergent strengths. ChatGPT-4o and Gemini achieved higher accuracy on formal grammatical tasks, whereas Claude and Grok struggled with cultural references and idioms. Across all systems, challenges persisted in idiomatic translation, negative polarity handling, and interpretation of culturally bound metaphors. Semantic preservation was moderately successful, but cultural sensitivity was frequently insufficient, especially in folklore-based and region-specific content. Inter-model variability was high, reflecting differences in training data and alignment. Conclusions LLMs show promising capabilities in processing Kurmanji, yet substantial limitations remain in producing culturally resonant and semantically precise outputs. These findings underline the need for more inclusive training corpora, culturally informed evaluation benchmarks, and community-driven assessments to advance LLM performance on low-resource languages.</ns3:p>