Simulated Morality, Misplaced Trust: The Risks of Treating AI as a Moral Partner

Abstract The article argues that many contemporary AI alignment practices risk a mistaken assimilation of moral agency to statistical learning. Techniques such as reinforcement learning from human feedback and constitutional AI often treat morality as a behavioral function that can be approximated from human discourse, behavior, and large-scale interaction data. Drawing on Tomasello’s account of shared intentionality, Hegel’s theory of recognition, and the second-personal tradition in moral philosophy, the article contends that moral agency is not exhausted by norm-conforming output, but presupposes participation in a space of mutual accountability, justificatory practice, and normative self-binding. On this view, and given the architecture of present generative systems, the production of normatively fluent behavior and reason-like discourse is not sufficient for moral partnership: absent interpersonal identity, recognition, and second-personal answerability, such systems can at best simulate the outer form of moral conduct. The central risk, therefore, is not that these systems behave “immorally,” but that their reliable norm-conforming behavior is misread as evidence of genuine moral commitment, encouraging misplaced trust and over-ascription of responsibility. The article proposes a reframing of alignment from an ethical-pedagogical project to an institutional one: instead of presuming or attempting to cultivate artificial moral agents on the basis of behavioral proxies, governance should focus on designing legal, technical, and organizational structures that constrain and render machine behavior auditable and accountable, while keeping moral responsibility firmly anchored in human agents and institutions.