ZharfSima: Benchmarking Vision Language Models for Persian Language and Culture

This literature presents ZharfSima. the first comprehensive benchmark for evaluating Vision-Language Models (VLMs) tailored to the Persian language and Iranian cultural contexts. The benchmark includes datasets for reasoning, Iranian cultural knowledge, standard and text generation, featuring tasks like driving test questions, IQ problems, and cultural assessments. Key contributions include the introduction of the Persian SEED benchmark and unique datasets like NaarDoon, designed for evaluating cultural and visual reasoning. By utilizing ZharfSima benchmark, we compared the performance of open-source VLMs (Qwen2- VL, Llama-3.2-Vision, NVLM-D) and GPT-40 on classification and text generation tasks. Results show GPT-40 excels in cultural understanding and text generation, achieving over 70% accuracy in Iranian Essence task, while Qwen2- VL performs best among open models. However, reasoning tasks remain challenging for all models, with performance under 30%. This work highlights the need for improved VLM training on low-resource languages like Persian and the impact of instruction's language on performance, where English questions outperform Persian ones. This benchmark serves as a critical resource for advancing multimodal AI in underrepresented languages.