Against Human Content Moderation: Algorithms without Trauma

ABSTRACT This paper explores the morality of human content moderation. It focuses on content moderation of Child Sexual Abuse Material (CSAM) as it takes place in commercial digital platforms, broadly understood. I select CSAM for examination because there is a widespread and uncontroversial consensus around the need to remove it, which furnishes the strongest possible argument for human content moderation. The paper makes the case that, even if we grant that social media platforms or chatbots are a valuable—or inevitable— force in current societies, and even if moderation plays an important role in protecting users and society more generally from the detrimental effects of these digital tools, it is far from clear that tasking humans to conduct such moderation is permissible without constraints, given the psychic toll of the practice on moderators. While a blanket prohibition of human moderation would be objectionable, given the benefits of the practice, it behooves us to identify the fundamental interests affected by the harms of human moderation, the obligations that platforms acquire to protect such interests, and the conditions under which their realization is in fact possible. I argue that the failure to comply with certain standards renders human moderation impermissible.