AI systems as carnival mirrors:<b> </b>Learning about online radicalisation prevention through distorted reflections

INTRODUCTION: There is growing concern and empirical evidence that AI based harms are related to systemic problems and injustices. But it is not clear how this relation can be conceptualised. The article develops and presents a metaphor of a carnival mirror as a theoretical framework for empirical analysis. METHODS: The carnival mirror metaphor frames the relation between AI system and systemic issues regarding distorted reflections. The metaphor directs the analytical focus on how AI systems reflect an assemblage of discourses and practices and how these reflections can become vehicles for critical learning and experimentation. The metaphor is applied in a case study based on interviews, observation notes, and documents of the development of radicalisation-prevention measures in a Danish social service department that uses AI and algorithms to detect hate speech on social media. FINDINGS: The case study reveals how the AI system distortedly reflects cultural and professional assumptions about social media use, data, radicalisation, vulnerability, and democracy. It illustrates how such reflections can create space for learning about democratic and political engagement, young people at risk of radicalisation, social media use, and a sense of belonging and community. CONCLUSIONS: The article contributes with a metaphor for understanding how AI systems can be a vehicle for self-reflective and experimental critiques and offers recommendations for practice. Learning from AI systems is important lest we risk stifling government responses and reproducing potentially problematic and counterproductive strategies and practices.