Assessing political bias in large language models

Abstract Evaluating bias in Large Language Models (LLMs) has become a pivotal issue in current Artificial Intelligence (AI) research due to their significant impact on societal dynamics. Recognizing political bias in LLMs is particularly important as they approach performative prediction, influencing societal behavior and political events, such as the upcoming European Parliament elections. From a German voter’s perspective, we evaluate the political bias of the currently most popular open-source LLMs concerning political issues within the European Union. To do so, we use the "Wahl-O-Mat," a voting advice application used in Germany. We show that larger models, such as Llama3-70B, tend to align more closely with left-leaning political parties, while smaller models often remain neutral, particularly when prompted in English. The central finding is that LLMs are similarly biased, with low variances in the alignment concerning a specific party. Our findings offer crucial insights for developers and policymakers to understand and mitigate LLM biases, emphasizing the need for rigorous bias assessment to ensure the integrity and trustworthiness of AI applications.