Navigating the Political Compass: Evaluating Multilingual LLMs across Languages and Nationalities

Large Language Models (LLMs) have become ubiquitous in today's technological landscape, boasting a plethora of applications and even endangering human jobs in complex and creative fields.One such field is journalism: LLMs are being used for summarization, generation, and even fact-checking.However, in today's political landscape, LLMs could accentuate tensions if they exhibit political bias.In this work, we evaluate the political bias of the 15 most-used multilingual LLMs via the Political Compass Test.We test different scenarios, where we vary the language of the prompt while also assigning a nationality to the model.We evaluate models on the 50 most populous countries and their official languages.Our results indicate that language has a strong influence on the political ideology displayed by a model.In addition, smaller models tend to display a more stable political ideology, i.e. ideology that is less affected by variations in the prompt.as the judge?a study on judgement bias.