Moral disagreement and the limits of AI value alignment: a dual challenge of epistemic justification and political legitimacy

AI systems are increasingly in a position to have deep and systemic impacts on human wellbeing. Projects in value alignment, a critical area of AI safety research, must ultimately aim to ensure that all those who stand to be affected by such systems have good reason to accept their outputs. This is especially challenging where AI systems are involved in making morally controversial decisions. In this paper, we consider three current approaches to value alignment: crowdsourcing, reinforcement learning from human feedback, and constitutional AI. We argue that all three fail to accommodate reasonable moral disagreement, since they provide neither good epistemic reasons nor good political reasons for accepting AI systems' morally controversial outputs. Since these appear to be the most promising approaches to value alignment currently on offer, we conclude that accommodating reasonable moral disagreement remains an open problem for AI safety, and we offer guidance for future research.