SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models’ Knowledge of Indian Culture

Language models (LMs) are indispensable tools shaping modern workflows, but their global effectiveness depends on understanding local socio-cultural contexts.To address this, we introduce SANSKRITI, a benchmark designed to evaluate language models' comprehension of India's rich cultural diversity.Comprising of 21,853 meticulously curated question-answer pairs spanning 28 states and 8 union territories, SANSKRITI is the largest dataset for testing Indian cultural knowledge.It covers sixteen key attributes of Indian culture namely rituals and ceremonies, history, tourism, cuisine, dance and music, costume, language, art, festivals, religion, medicine, transport, sports, nightlife and personalities, providing a comprehensive representation of India's cultural tapestry.We evaluate SAN-SKRITI on leading Large Language Models (LLMs), Indic Language Models (ILMs), and Small Language Models (SLMs), revealing significant disparities in their ability to handle culturally nuanced queries, with many models struggling in region-specific contexts.By offering an extensive, culturally rich, and diverse dataset, SANSKRITI sets a new standard for assessing and improving the cultural understanding of LMs.societies.We developed the SANSKRITI dataset to benchmark cultural understanding in India, utilizing diverse data sources to cover key attributes like rituals and ceremonies, history, tourism, cuisine, dance and music, costume, language, art, festivals, religion, medicine, transport, sports, nightlife, and personalities.A team of 40 annotators curated 21,853 questions across four categories: association, country prediction, general awareness, and state prediction.Research Questions: This research aims to address the following questions:1. How do various language models (LLMs, SLMs, and ILMs) perform on the SANSKRITI dataset, and what are the performance trends across these model categories?2. How do language models perform across different cultural attributes represented in the SAN-SKRITI dataset?3. What are the performance trends of language models across different Indian states and union territories?4. How do language models perform across various question types in the SANSKRITI dataset?