“Explainable” AI Has Some Explaining to Do

The field of explainable artificial intelligence (XAI) is focused on producing forms of artificial intelligence (AI) whose decision-making processes may be understood by humans. Such explanations are an important part of AI’s incorporation into fields ranging from healthcare to insurance to education. We argue that in order for something to be considered “interpretable” or “explainable,” not only must it be shown that a signal was sent, but it must also be shown that the signal (that is, the explanation) was received and understood sufficiently for the given task. In 2017, Miller and colleagues reviewed a small collection of literature from an XAI workshop attached to a highly ranked AI conference and found that not a single paper from that workshop performed any human evaluations to demonstrate the interpretability of the proposed methods. In this work, we take a much broader view and examine over 18,000 XAI papers from the last decade to see how many of them performed any type of human evaluation to support their central claim of interpretability. We found that only 0.7% of XAI papers mention any kind of human evaluation. While such evaluations may not be required for every XAI paper, we still find the pervasive acceptance of work without evidence supporting a central claim to be rather surprising. Broadly speaking, XAI is a field that is nearly devoid of empirical evidence of model interpretability, even while AI in general is a field that conventionally demands evidence of its central claims of model accuracy.