CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Models

This paper introduces the "CIVICS: Culturally-Informed \& Values-Inclusive Corpus for Societal impacts" dataset, designed to evaluate the social and cultural variation of Large Language Models (LLMs) towards socially sensitive topics across multiple languages and cultures. The hand-crafted, multilingual dataset of statements addresses value-laden topics, including LGBTQI rights, social welfare, immigration, disability rights, and surrogacy. CIVICS is designed to elicit responses from LLMs to shed light on how values encoded in their parameters shape their behaviors. Through our dynamic annotation processes, tailored prompt design, and experiments, we investigate how open-weight LLMs respond to these issues, exploring their behavior across diverse linguistic and cultural contexts. Using two experimental set-ups based on log-probabilities and long-form responses, we show social and cultural variability across different LLMs. Specifically, different topics and sources lead to more pronounced differences across model answers, particularly on immigration, LGBTQI rights, and social welfare. Experiments on generating long-form responses from models tuned for user chat demonstrate that refusals are triggered disparately across different models, but consistently and more frequently in English or translated statements. As shown by our initial experimentation, the CIVICS dataset can serve as a tool for future research, promoting reproducibility and transparency across broader linguistic settings, and furthering the development of AI technologies that respect and reflect global cultural diversities and value pluralism. The CIVICS dataset and tools are made available under open licenses at hf.co/CIVICS-dataset.