The AI Referee: How Online Interventions Shape Incivility and User Engagement in News Discussions

This paper seeks to understand how online interventions shape incivility and user engagement with news comments. Using a novel dataset of over 39 million news comments on Korea's largest online news source (Naver News), we examine changes in the share of comments that are categorized as uncivil before and after the introduction of two automated interventions aimed at flagging incivility. We trained two deep learning models to categorize comments and replicate each intervention. The ?findings reveal signifi?cant decreases in uncivil content following each intervention. Interestingly, we find mixed effects of the interventions on total engagement: while the numbers of comments and commenters decreased after the ?rirst intervention, both metrics increased after the second. Examining individual level data reveals that the aggregate reduction in incivility cuts across all users regardless of pre-intervention incivility or commenting frequency.