AI systems should be trustworthy, not trusted

Abstract Scientific and public debates on the ethical aspects of AI development and deployment often end up focusing on trust in AI systems, rather than on their trustworthiness. This paper argues that actual trust should not be the focus of the debate in AI ethics or the goal of the responsible design, deployment, and assessment of AI systems. The argument will insist on three distinct—although interrelated—points. First, I will argue that trust is a complex psychological phenomenon that is influenced by many contextual and non-rational factors that may have little to do with AI systems’ actual trustworthiness. Then, I will show that some widely employed strategies to foster trust in AI are ethically questionable and hardly compatible with the trustworthy AI paradigm. Finally, I will focus on the fact that trust might lead to unmonitored reliance on systems whose risks are not negligible and, in many cases, largely unknown.