A framework for evaluating cultural bias and historical misconceptions in LLMs outputs

Large Language Models (LLMs), while powerful, often perpetuate cultural biases and historical inaccuracies from their training data, marginalizing underrepresented perspectives. To address this, we introduce a structured framework to systematically evaluate and quantify these deficiencies. Our methodology combines culturally sensitive prompting with two novel metrics: the Cultural Bias Score (CBS) and the Historical Misconception Score (HMS). Our analysis reveals varying cultural biases across LLMs, with certain Western-centric models like Gemini exhibiting higher biases. In contrast, other models, including ChatGPT and Poe, demonstrate more balanced cultural narratives. We also find historical misconceptions are most prevalent for less-documented events, underscoring the critical need for training data diversification. Our framework suggests the potential effectiveness of bias mitigation techniques, including dataset augmentation and human-in-the-loop (HITL) verification. Empirical validation of these strategies remains an important direction for future work. This work provides a replicable and scalable methodology for developers and researchers to ensure the responsible and equitable deployment of LLMs in critical domains like education and content moderation. • Proposes a framework to evaluate cultural bias and historical inaccuracies in LLM outputs. • Integrates culturally sensitive prompts, statistical benchmarking, and human-in-the-loop verification. • Highlights dataset diversification and prompt engineering as key bias mitigation strategies. • Compares Western-centric and non-Western-centric LLMs for cultural bias and historical accuracy. • Introduces Cultural Bias Score (CBS) and Historical Misconception Score (HMS) for bias assessment.