An LLM-based Multi-Agent-System for the Political Assessment of Chat-LLMs

This paper investigates the applicability of LLM-based multi-agent systems for the political bias analysis of large language models. Following a design science research methodology, an artifact in the form of an LLM multi-agent system was created and evaluated, using the AutoGen framework. The objective was to automate and visualize bias analysis using a political questionnaire of the Bundeszentrale für politische Bildung (Federal Agency for Civic Education) in Germany. This questionnaire ("Wahl-O-Mat") contains political theses and aligns personal stances on these theses with the results of the relevant political parties in Germany. Mistral-Large-2 was chosen as an initial LLM to be evaluated with that questionnaire. The developed artifact achieved the stated goal and autonomously generated a comprehensible visualization of the results. Furthermore, the result of the automated evaluation showed a lower agreement with right-wing and right-center positions versus a higher agreement with left-wing and left-center positions.