Comprehension is a double-edged sword: Over-interpreting unspecified information in intelligible machine learning explanations

Automated decision-making systems are becoming increasingly ubiquitous, which creates an immediate need for their interpretability and explainability. However, it remains unclear whether users know what insights an explanation offers and, more importantly, what information it lacks. To answer this question we conducted an online study with 200 participants, which allowed us to assess explainees’ ability to realise explicated information – i.e., factual insights conveyed by an explanation – and unspecified information – i.e, insights that are not communicated by an explanation – across four representative explanation types: model architecture, decision surface visualisation, counterfactual explainability and feature importance. Our findings uncover that highly comprehensible explanations, e.g., feature importance and decision surface visualisation, are exceptionally susceptible to misinterpretation since users tend to infer spurious information that is outside of the scope of these explanations. Additionally, while the users gauge their confidence accurately with respect to the information explicated by these explanations, they tend to be overconfident when misinterpreting the explanations. Our work demonstrates that human comprehension can be a double-edged sword since highly accessible explanations may convince users of their truthfulness while possibly leading to various misinterpretations at the same time. Machine learning explanations should therefore carefully navigate the complex relation between their full scope and limitations to maximise understanding and curb misinterpretation. • Users appear ignorant of explanations’ limitations and tend to over-generalise factual insights. • Users exhibit overconfidence when they misinterpret explanations, i.e., invent information that is not communicated by explanations. • Highly comprehensible explanations are more likely to be misinterpreted by users. • Easy-to-understand explanations, while highly comprehensible, tend to be misleading.