Design for operator contestability: control over autonomous systems by introducing defeaters

Abstract This paper introduces the concept of Operator Contestability in AI systems: the principle that those overseeing AI systems (operators) must have the necessary control to be accountable for the decisions made by these algorithms. We argue that designers have a duty to ensure operator contestability. We demonstrate how this duty can be fulfilled by applying the'Design for Defeaters' framework, which provides strategies to embed tools within AI systems that enable operators to challenge decisions. Defeaters are designed to contest either the justification for the AI’s data inputs (undercutting defeaters) or the validity of the conclusions drawn from that data (rebutting defeaters). To illustrate the necessity and application of this framework, we examine case studies such as AI-driven recruitment processes, where operators need tools and authority to uncover and address potential biases, and autonomous driving systems, where real-time decision-making is crucial. The paper argues that operator contestability requires ensuring that operators have (1) epistemic access to the relevant normative reasons and (2) the authority and cognitive capacity to act on these defeaters. By addressing these challenges, the paper emphasizes the importance of designing AI systems in a way that enables operators to effectively contest AI decisions, thereby ensuring that the appropriate individuals can take responsibility for the outcomes of human-AI interactions.