Legitimate Power, Illegitimate Automation: The problem of ignoring legitimacy in automated decision systems

Progress in machine learning and artificial intelligence has spurred the widespread adoption of automated decision systems (ADS). An extensive literature explores what conditions must be met for these systems’ decisions to be fair. However, questions of legitimacy—why those in control of ADS are entitled to make such decisions—have received comparatively little attention. This article shows that when such questions are raised theorists often incorrectly conflate legitimacy with either public acceptance or other substantive values such as fairness, accuracy, expertise or efficiency. In search of better theories, we conduct a critical analysis of the philosophical literature on the legitimacy of the state, focusing on consent, public reason, and democratic authorisation. This analysis reveals that these theories require careful consideration before being applied to legitimate ADS. Specifically, we find that consent and public reason theories can legitimate some automated decision systems, but encounter serious difficulties which limit their applicability. Democratic authorisation, though not without its own challenges, emerges as a more promising and broadly applicable means of legitimation.