Cultural Evaluation of LLMs in Russian: Catchphrases and Cultural Types

This study addresses the gap in evaluating large language models' (LLMs) cultural awareness and alignment within the Russian sociocultural context by introducing a structured framework comprising 8 Cultural Types (e.g., Spiritual Practitioner, Soviet Intellectual) and 5 catchphrase groups (e.g., memes, proverbs).A 400-question evaluation dataset was developed to probe 10 multilingual LLMs, including GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro, across fact-based cultural knowledge and nuanced linguacultural understanding in a zero-shot setting.Results show that closed-source models GPT-4o and Claude 3.5 Sonnet outperform other models, with one of the smallest models (Mistral NeMo 12B) achieving the lowest result.Performance disparities were noticed in separate evaluation on Cultural Type tasks and catchphrases.Model-specific skews emerged, with lower-ranked models showing inclination toward specific cultural types.Qualitative analysis revealed common errors, such as selecting synonymous but incorrect answers or failing to grasp culturally specific logic.The contribution outlines the limitations of LLMs in interpreting cultural context and lays the groundwork for further research in assessing the cultural-linguistic alignment of LLMs.