Algorithmic Trust and Agential Possession

Abstract Research in AI safety and AI ethics tends to focus on two types of ethical danger: unethical results of AI usage—such as misinformation—and unethical side‐effects from AI usage—such as environmental degradation. These dangers are worthy of consideration; however, the literature has not adequately considered non‐consequentialist dangers (such as moral wrongs), which might be intrinsic to the relationship between human agents and AI tools. I argue that by contrasting the structure of human agency with algorithmic agency we can identify ethical dangers grounded in a mismatched relationship between human agents and AI agents. Moreover, I claim that by analyzing how humans trust AI agency we reveal an understudied threat, what I call the “diabolical exchange,” which emerges when human agents conform to the structure of merely functional AI‐agents. I conclude that since current studies in AI safety and AI ethics do not yet have the conceptual resources to fully articulate this agential wrong, theology may be able to help. This is because the ethical danger intrinsic to the human‐to‐AI relationship is best captured by analogy to the theological concept of demonic possession as found in early Christian religious texts. I end by briefly considering what the concept of possession might teach us about ethical responsibility under non‐ideal instrumentarian conditions.