Explanation needs and ethical demands: unpacking the instrumental value of XAI

Abstract The call for XAI rests on a normative claim: ‘Good AI is explainable AI’ or even the stronger claim: ‘Only explainable AI is good AI.’ However, this valorization runs the risk of being overgeneralized because explanations are not per se useful, appropriate, or demanded. Explainability should not be seen as a value in itself but as a means to certain ends. In this paper, we put the valorization of explainability into question, which is discursively connected to the idea of ‘users’ needs’ and the will to design and develop ethically aligned AI systems. By making the instrumental character of the value of explainability explicit, we address two key issues that necessitate more theoretical attention: (i) to analyze the link between explainability and its presumed purpose; and (ii) to clarify the conceptions of these presumed purposes, namely users’ needs and ethical principles XAI is meant to promote. From a philosophical and from a psychological perspective, we constructively criticize the undertheorized and undercomplex way of talking about ‘users’ needs’ and ethical demands. We plea to carefully differentiate the value of explainable AI in social contexts and signal further need for research.